Deep Learning Full Course — Step-by-Step Guide & Transcript

[00:02] patterns, make prediction and solve real world problems almost like a human learning. Hello everyone and welcome back to simply learn. In this video we will explore the complete beginner to practical journey of deep learning. We

[00:15] will understand how neural networks work, how machines learn from the data and how frameworks like TensorFlow and PyTorch help us build real deep learning models. This topic is important because deep learning is behind many of today's

[00:28] most powerful AI applications including image recognitions, speech assistance, recommendation systems, natural language processing, self-driving cars, and even strong foundation in AI and machine learning, deep learning is one of the

[00:42] session, we will cover the basics of deep learning, artificial neurons, perceptrons, weights, activation function, forward progression, loss function, gradient descent and back propagation. We will also move into

[00:56] practical implementation using TensorFlow, Keras and PyTorch. You will see how neural networks are built, trained, evaluated and applied to real classification problems using data sets like fashion and a capstone style

[01:09] you'll understand how deep learning models learn, how training works, and how to start building your own neural networks using modern deep learning frameworks. Before we move ahead, let me quickly share you something exciting for

[01:22] career in generative AI and machine learning. The ENICT Academy IIT Kpur professional certificate course in generative AI and machine learning is designed to help you learn the complete AI and ML skill set from Python, data

[01:37] science, machine learning, deep learning and NLP to generative AI, prompt engineering, LLMS, computer vision and reinforcement learning. The program includes live online classes, master classes designed by IIT Kpur faculty and

[01:51] practical exposure to popular tools like chat GPT, hugging face, DALE, TensorFlow, Keras, Graddio, Lang Chain, OpenAI and more. You'll also work on 15 plus hands-on projects covering real world use cases like employee attrition

[02:05] prediction, loan default analysis, AI powered HR assistance, texttoes design platform, recommendation engines, sales forecasting and deep learning applications. On completing the program, you'll receive a program completion

[02:18] certificate from ENIC Academy, IIT Kpur along with official Microsoft course badges for eligible learning path. So if you want to go beyond just using AI generative AI and machine learning solutions are built and applied in real

[02:33] worth checking out. The link is given in the description box below and in the pin continue, here's a quick question for you to answer. Which of the following is commonly used as an activation function in hidden layers of neural networks? Is

[02:47] in hidden layers of neural networks? Is it RLU, HTML, SQL, or Excel? Let me know below. Let's get started. I I've always found that in in teaching this several times uh in this program getting to the deep learning is um challenging doing

[03:04] the deep learning is challenging but very rewarding and I think a lot of people enjoy the topics here because they're very u modern and very relevant they're very u modern and very relevant to um very advanced kind of modeling uh

[03:18] techniques. So, a lot of people enjoy it, but it is going to be a little bit more challenging than what we've done so far. So, that's okay. I think we can handle it. Um, so yeah, let's start with our

[03:32] introduction here. Okay. So, what we're hoping to here. Okay. So, what we're hoping to cover here are just the uh some of the information about deep learning. So some of the um achievements that have really

[03:46] of the um achievements that have really happened in recent times um and we're talking you know less than 10 years old and some of these achievements that are uh have brought deep learning to the forefront and made it a really popular

[04:00] uh field. So we'll understand some of the applications and also some of the deep learning is not perfect. It has some drawbacks and some challenges that make it um difficult at times. So we'll talk about that and then we'll talk

[04:13] already mentioned uh PyTorch and TensorFlow as the two main ones. Just talk about that a little bit. Um and then talk about what are the main steps of any deep learning project that uh we would want to follow. I think what

[04:26] you're going to see there is they mostly mimic the same kind of steps we were doing in machine learning. things like uh data preparation, then kind of building a model, training it, um evaluating it on test data. Those kind

[04:41] of steps are mostly the same. It's just the details of those are going to be training process is going to be more involved, I think, with with deep learning. Um building the model is a little more hands-on with deep learning

[04:54] as well, cuz you're literally going to be architecting and constructing a neural network. It's much different than just instantiating a um a model from scikitlearn which is pretty easy to do. Um there's going to be more hands-on

[05:08] work involved in in building these deep learning uh networks that are going to be our models. Um so we'll talk about that process. Okay. All right. So let's start with just AI in general. So the reason to start here is just to see

[05:22] where deep learning kind of fits into the rest of the stuff that we've studied so far such as machine learning and um historically historically uh where has deep learning fit in. So, I

[05:37] early on in our machine learning and even data science uh uh courses, but even data science uh uh courses, but here it is again um where we have AI as here it is again um where we have AI as this pretty broad umbrella um that is um

[05:53] basically involving machines, so computers um trying to perform tasks that normally humans would have to do. And what we just got finished studying was machine learning, right? So machine learning is primarily having the machine

[06:09] learn from data how to make a decision right how to complete some task whether that's prediction or maybe like an unsupervised case um doing things like compression dimensionality reduction or finding patterns and clusters it's

[06:23] learning from data how to do that right that's machine learning broadly um where that's machine learning broadly um where deep learning fits in is really a subset of machine learning because We're still learning from data. Uh we still are

[06:38] doing that. It's just we're using a specific tool to do that which is going to be a neural network. Um so that's why in this diagram you see deep learning as kind of a circle embedded inside of the machine learning circle because it's it

[06:52] is kind of a sub uh field of machine data. You're just doing it in a very particular way. Um, so it's kind of like a specialization. That's why I said earlier not everybody

[07:07] get pretty far without deep learning. You still have this whole field of machine learning and you can you can still do a lot with that. Um, but this is certainly kind of a specialization is drilling down into working with neural

[07:22] nets in particular as your model. It turns out that working with neural nets turns out that working with neural nets is really powerful for many reasons. Um, and what we're going to see is that deep learning really unlocks uh working with

[07:35] specific types of data that we have not worked with so far. And you know, we already have a sense of that. Things like images, things like text, we really haven't worked with too much or at all. Um, but deep learning really unlocks the

[07:49] ability to do that. Um, so you can see that here. Yeah, there there are um we've briefly talked about a couple of them. Uh so inside of machine learning that is um not necessarily deep learning would be something like RL. So and it's

[08:04] not really a small circle but probably another big one is uh and it it honestly it kind of overlaps with deep learning because you can use uh neural networks inside of reinforcement learning. So RL is short for reinforcement learning. I

[08:17] draw it overlap just to say that nowadays a lot of RL is done with deep learning ideas in terms of using neural nets but it wasn't always that way but nets but it wasn't always that way but yeah there's RL and then there's uh

[08:32] certainly like a lot of use cases of unsupervised learning inside of machine learning that does not use neural nets. So unsupervised learning. So things like clustering uh unsupervised learning, you know,

[08:49] thing basically things that do not use neural nets but are but are still you're still learning from data. Yeah, it's basically everything we covered in the last course. Um so even supervised learning that's not using neural nets is

[09:02] still in this in this bucket of machine learning. All right. So, so this is this learning. We're going to be in this course. This is where it kind of fits learning because at the end of the day, we're still learning from data. It's

[09:16] just going to be using a particular type of model, which is a neural net. And more so than just a basic neural net, it's often going to be what we will um term as a deep neural net. meaning it will be a pretty complex model that has

[09:31] will be a pretty complex model that has uh a lot of layers to it um as we will see but it is just a subset of machine learning.

[09:44] evolved to get to that point? Um well we had originally the the very first model of a neuron was proposed in 1943 um by mulla and pitts. So they produced

[09:58] a paper kind of um proposing a potential model of the of the brain and the model of the of the brain and the neurons in the brain uh in 1943. So obviously a long time ago. And at the same time some other things going on. We

[10:11] had the Turing test which you may have heard of basically um testing the heard of basically um testing the intelligent ability of a machine. And so we had that going on as well. Obviously other things going on from the 40s to

[10:24] 50. But the big one that's relevant to deep learning is the very first proposed model of the neuron that was as old as 1943. It was more of an idea then. Obviously they didn't have computing power to do

[10:39] anything with that. Um it was more just the idea. The concept was kind of as old as that of just modeling an actual neuron. Um it wasn't until the 50s that we see the actual um term artificial intelligence being adopted.

[10:56] intelligence being adopted. Um as well as the very first very basic neur uh neural network model the perceptron uh that was invented in 1957. At the same time we have some

[11:09] programming languages being invented uh to take advantage of machines. um not to take advantage of machines. um not Python yet obviously but you know some older languages that end up inspiring the uh generation of Python later on but

[11:25] still some some uh highle languages that are invented but this perceptron is really relevant actually we're going to study perceptron in the next lesson as study perceptron in the next lesson as kind of the first model the first basic

[11:39] neural network model that uh will propel you know uh deeper and deeper networks uh built off of the idea of kind of a perceptron. So we'll talk about perceptron quite a bit. Okay. So then we go to the 60s are perceptron still being

[11:57] go to the 60s are perceptron still being used today. Um perceptron itself is probably not used too much but the idea of the perceptron is very present in so many network. it. Think of it like the building block of any neural network um

[12:13] building block of any neural network um was founded on the perceptron idea. So the the perceptron itself like just a single neuron. So so we're going to learn about this but the perceptron is basically the model of a single neuron.

[12:25] A single neuron not quite used too much. Uh it's very basic but it forms the building block for nearly all neural networks that are around today. So I

[12:37] would say yeah I would say it's it's inspired pretty much any neural network is inspired by the perceptron. So it's a very very important idea and concept and we're going to spend some time talking

[12:49] about it and studying it uh as kind of our foundation for neural nets and then we'll build on that by kind of stacking uh stacking multiple neural multiple perceptrons essentially together uh till we get a larger and larger network.

[13:06] Yeah. So the thing is the idea has been around a long time. Uh but there hasn't been the you know computing power to really take advantage of it until recently. And when I say recently probably last 15 20 years um hasn't been

[13:21] right there hasn't been the computing power available to really unlock the capabilities of these and make make uh larger and larger networks that can solve harder and harder problems. So we don't see that uh we don't see that

[13:35] until you know recent first job was a key punch operator. I think it's the uh don't know what key punch is. I think it's the um like the binary uh the bit

[13:47] uh cards that uh fed the original kind of computers or older computers and you of computers or older computers and you had to literally punch out the bits. Also this time in the 60s was the original chatbot Eliza um 1966. Now it's

[14:05] nothing like generative AI right this is this chatbot is all rules driven so nothing data driven nothing like that certainly no neural net nothing that we see today with these transformerbased models and GPTs and such but it is a

[14:22] text and then it can respond with text it had to be pre-programmed uh how to respond to things but that was uh how to respond to things but that was you know the first kind of AI chatbot

[14:36] Okay. So then we see kind of a uh lull Okay. So then we see kind of a uh lull in the uh 70s to 8 70 to 80. Um interest in AI dropped significantly. Um funding from from government sources was

[14:49] insufficient. So there's kind of a lull period here. Now that's not to say that um there wasn't anything going on. You know, there's still statistical research going on and and advancements in the machine learning community, but nothing

[15:04] machine learning community, but nothing really relevant to neural networks uh at this period. But still, you know, some of the some of the developments in the in the statistics community are still relevant during this time uh to machine

[15:18] learning, just not necessarily deep learning. But there's a rebound of that in the 80s. Um, and you may be able to guess 80s. Um, and you may be able to guess why. Maybe due to geopolitical things

[15:33] going on in the ' 80s, like the cold war. Um, we see a revival in uh research war. Um, we see a revival in uh research and dedication to AI. Um, we also start to see some communities uh develop um that dedicated to the advancement of AI.

[15:50] So the Stanford hosts the inaugural uh conference the AAI conference the AAI um which is kind of the uh inspiration for some of the conferences we see today like nuripss and and things like that.

[16:03] So in the 80s there's kind of a revival and then there's a kind of a um interesting period of time in the late late 90s so

[16:16] kind of early I would say even early 90s to early 2000s in this period um we see to early 2000s in this period um we see a significant investment in AI from the big players like like Facebooks and Netflixes of the world and Google of

[16:31] course Twitter they started investing in AI quite a bit. 1997 we have the IBM deep blue if you've ever heard of that. Um beat the uh world chess champion. Um

[16:43] so that was a really cool kind of AI achievement. But this is this period's called out here in particular because AI is particular deep learning had not advanced that far even by this point.

[16:58] However, there are some significant investments in this period that lead us to kind of where we are today. Um, I would say two of them that are not necessarily listed here but will be coming up are where it says companies

[17:11] like Facebook, Twitter and Netflix and Google not mentioned but should be there. They really start investing in AI. I want to call out Facebook and Google in particular because they build the frameworks that we use today. So

[17:27] Facebook invented PyTorch around this time and Google invented TensorFlow around this time. They did those independently. Now they are mostly open

[17:40] sourced so anybody can use them but they were in-house tools to work with neural nets. Everybody wanted to work with neural nets. They had their own in-house tools uh like PyTorch and TensorFlow to kind of build and manage neural nets at

[17:53] the time. And this all started right around this period 2006 uh or so you know not that long ago relative to uh history only about 20 years not that long ago and we see kind of those foundational uh libraries being

[18:08] developed and obviously there's been a lot of advancements since then. So in uh lot of advancements since then. So in uh the period after that um this is kind of where the deep learning golden age is like really the 2010s onward 2012 is

[18:23] about there's a really big event in 2012 that happens which is the Alex net that happens which is the Alex net convolutional net is is uh is published and studied it's a really big computer vision model so a lot of really amazing

[18:39] achievements from from this period really 2010s onward in deep learning. And it kind of coincides with hardware being more available, um, neural nets

[18:51] being more readily available with those frameworks being more mature and developed like PyTorch and TensorFlow. Um, more people have access to them. They're open source, so more people can work with them and and really get their

[19:04] hands on it, including the research community. So, there's a reason it kind of blows up in the 2010s onward. [snorts] even even today. Uh it's so I'll tell you why. It's because um a lot of generative AI is powered by the

[19:17] of generative AI is powered by the transformer and the transformer was not transformer and the transformer was not developed really until 2017. developed really until 2017. So that's less than 10 years ago. Um but

[19:29] the really the computing power and data availability to work with something like a transformer has not caught up until really like the last five or so years. So that's why it's so recent. The model that they're all built off of is less

[19:43] that they're all built off of is less than 10 years old. Um the transformer, but it's more like just other things like hardware and data availability. like hardware and data availability. We've caught up to that. uh and and now

[19:57] we can have these really massive language models that power generative AI because compute power is relatively easy to to manage and um data is pretty easy

[20:09] to get your hands on too but even but that's what I'm saying the model is not that old that powers it all right but a really good question um so what are some of our motivations for deep learning so

[20:22] there's a number of topics that we're going study, you know, they're all listed here. We're going to start with the perceptron. We're going to to talk about these topics later on that have to do with like uh the back propagation

[20:36] really having to deal with how to train a neural net. Um so we'll talk about how how they get trained. We'll talk about the earliest model of a neuron which is going to be the perceptron. So we'll certainly cover that. Um then

[20:51] we can stack perceptrons together in what what is known as a multi-layer perceptron. That's kind of the next level. Um that's kind of like a miniature deep neural net. It's it's not very deep, but it does have multiple

[21:04] very deep, but it does have multiple layers to it. Um and then we have the kind of in more recent times these very advanced networks like these that we're going to study. um convolutions mainly being used for images and recurrent

[21:18] text. So we're talking about the motivation one of the motivations one of the primary motivations for talking about deep learning is really this this idea that

[21:33] ability to work with text data and our ability to work with image data. There are so many problems out there that work with this kind of data which in the community is known as unstructured data. Right? So, so far we've really only done

[21:50] problems with structured data. And structured meaning that it's basically a matrix that has rows and columns with really well-defined features, right? That are rows and columns like our like our housing data that has square footage

[22:04] and bedrooms and bathrooms. That's all very well structured. Text is not structured at all. It can be variable length. You can have paragraphs. You can have whole documents. It's unstructured. Um images, same thing. They can be many

[22:17] little bit more structure to them in the sense that they are matrices somewhat, right? They have um rows and columns of pixels, but still they're considered pixels, but still they're considered unstructured data. Um and and uh so

[22:33] neural nets and deep learning will unlock the ability to work with those. There's so many problems that work with images like object detection, image recognition, you know, and that extends to so many applications. And then

[22:47] working with text, obviously generative AI, working with text or even doing things like sentiment analysis or language translation, a lot of things work with text. So we need these advanced neural nets to be able to deal

[22:59] with that kind of data. The the models that we've studied so far just are not capable of working with that data very effectively at all. Like logistic regressions or decision trees or even things like an XG boost, they're going

[23:13] They're not really going to be able to process it. Neural net on the other hand is going to be able to. That's one of our motivations for studying it is it unlocks this whole set of applications for us that we couldn't do otherwise

[23:29] namely working with text and image data. Right? Okay. So some particular tasks there that we were that we are going to study. So

[23:41] things like image recognition. Um now the very basic example that I probably have said before but you know something like being able to predict if an image is a cat or a dog or if an image is um a cat, dog, a giraffe or a wolf or

[23:56] whatever it is, right? So image recognition tasks and that really extends into so many applications like object detection. So working with images really huge deal. We're going to devote a lot of time to it. Um but we really

[24:12] need neural nets to be able to do that effectively as we are going to find out. Um things with language like understanding and processing human language. We have these transformers that do this so effectively but it

[24:27] wasn't always that way and it was nearly impossible to do um with basic uh models. Just nearly impossible to understand language at the level that we see today. If you think about the the amount of language understanding it

[24:41] amount of language understanding it takes to have these um things like GPT and cla and stuff it's just immense and they're but they're really huge neural networks that are doing that and they're learning against so much data but we

[24:54] really need deep learning for that um and other applications things like speech recognition you know we haven't talked about it but another type of talked about it but another type of unstructured data is um audio right so

[25:06] audio data being able able to do things like recognition of audio translation like recognition of audio translation into text. That's a really hard problem. Basically impossible without neural nets uh to be able to recognize speech and

[25:21] audio. Same thing with video. I know it's not listed here, but video certainly unstructured. We need uh neural nets to be able to process those unstructured data we're going to use

[25:36] neural nets to be able to process them. Yeah. A lot of a lot of uh neural nets. So speech recognition is powered by we're going to cover this later with much later on in this course when we get into RNN's. Um a lot of things like um

[25:52] Alexa devices for instance have been powered by RNN neural networks to uh basically recognize your commands your voice commands. That's all neural recognize a voice command, translate

[26:06] text for creating uh tasks or doing things or or generating new text to be able to respond. Yeah, they're that is yeah they're neural nets. So this speech

[26:20] they're neural nets. So this speech recognition part is all um neural net driven namely RNN's recurrent neural nets um and and in modern times RNN/transformers we see transformers kind of taking over

[26:34] this space as well a lot of language stuff is being taken over by transformers but traditionally they've been RNN's okay so um let me give you

[26:47] some breakthroughs so for example You know, we talked about this period of know, we talked about this period of 2010s. One of the uh breakthroughs in 2010s. One of the uh breakthroughs in deep learning was in 2012, the Alex Net

[26:59] paper, which I encourage you to go look up and and read a little bit about. So, so the Alex net is a type of convolutional neural net and it convolutional neural net and it introduced really the convolution as a

[27:13] preeminent operation on image data to be able to classify images into different able to classify images into different categories. Um so it it this network architecture using convolutions um blew away the competition in the

[27:29] imageet challenge which was a um basically taking a bunch of images. So you can see here like the the training data has 1.2 million images testing data has more has 1.5 million images. Your goal is to categorize those images into

[27:45] 1,000 different categories. So, so think of it as like 1,000 different animals, not just cat or dog, but a thousand different uh animals. And um the best different uh animals. And um the best methods at the time um

[28:00] things like basic neural nets or maybe even like an XG boost were blown away by this convolutional net um by more than 10% which is astronomical amount in in 10% which is astronomical amount in in research um of this at this kind of

[28:15] level. So this was a huge breakthrough and what we see from there on is convolutions becoming kind of the default building block of anything detection, it's all going to use convolutions and it was all inspired by

[28:31] this Alex Snap paper. And it was for the first time really like people realizing like hey these neural nets can really work on unstructured data very

[28:45] effectively because the methodologies to work on these images prior to this was to basically try to generate a bunch of structured features which was not very

[28:57] effective. going from like an image into a flattened vector and trying to generate features out of that to pass through like an XG boost um was the through like an XG boost um was the strategy but it got surpassed immensely

[29:11] by this convolutional net and then we see just an explosion of computer vision progress with people using uh people using uh convolutions. So, one of the

[29:24] things we're going to study in this course is convolutions. They're really important for computer vision. If you're ever going to do anything with images, uh, any kind of neural nets, processing images is going to use convolutions in

[29:38] some fashion, right, in the architecture. So, we'll devote some time to studying convolutions for sure. But this was a huge breakthrough at the time. And it again, 2012, that's really not that long ago. It's it seems it's

[29:51] hard to believe that this was 2012. Um and you know that is kind of a long time somewhat but really not really not really in historical terms. Yeah, please check those out. I think the Alex net is really fascinating. Now it when you read

[30:06] that stuff it may not make any sense to you right now. That's okay. It should make a lot more sense by the time we get into uh computer vision and start studying convolutions. I think it'll make a lot more sense. So maybe hold on

[30:19] to those. I encourage you to read them, but they may not make any sense right now because we haven't covered anything about neural nets really um or especially convolutions. So they probably won't make any sense to you,

[30:31] but um I promise they will they'll make a lot more sense by the time we go through and talk about convolutions. Okay. So you know, we'll talk about some more breakthroughs. That's just one of them. Uh however there are challenges uh

[30:45] them. Uh however there are challenges uh that were encountered along the way. Um obviously like a lack of research funding was a big challenge. Um probably bigger like bigger than that definitely are these next two which are data and

[30:58] hardware. These are the two things that stalled deep learning progress for a long time. hardware mostly because neural nets and their structure, this is something we're going to learn about, benefit immensely from advanced GPU

[31:13] hardware. Um, it's primarily because the computations that a neural net does benefit a lot from the parallel parallelization that a GPU can provide at the hardware level. And so only until recently have we had a lot of

[31:27] advancements in hardware such that GPUs are readily available. Wasn't always that case. And also it wasn't always the case that that the code the Python code could take advantage of a GPU but it it can today pretty easily um with our

[31:44] frameworks like PyTorch and TensorFlow they they make it easy to take advantage of the GPU hardware and so that's why we've seen an explosion in the last 10 15 years because GPUs are more readily available and can be taken advantage of

[31:57] by the code. Um same thing with data. data is more readily available, right? So there's a lot more of it, a lot more it's it's cheaper to store it, a lot easier to collect it. So it wasn't always that case. And what the cases

[32:11] with neural nets is you need a lot of data to train a neural net more than what we've used so far for the structure data problems um that we've dealt with.

[32:23] data problems um that we've dealt with. Um, so the the the expensive nature of storing data and getting access to it was also prohibitive to a lot of progress in deep learning. But that's that's been kind of fixed, right? It's

[32:36] now a lot cheaper than ever. And consequently, we've seen a big explosion, right? Last 10-15 years. Okay. All right. Any questions? Uh, so far just some background, right? Just some background and a little historical

[32:49] some background and a little historical context. um haven't gotten too too much point of this first lesson, just more background information in historical context and then we'll start studying the models of the neuron and working our

[33:03] way up to you know more complex neural nets. Okay, let's talk about deep learning then uh a little more in particular. So we just covered the fact that um deep learning is really a subset of machine learning focusing on using

[33:18] neural nets and often what are going to be deep neural nets which are going to be deep neural nets which are going to be neural nets with uh a lot of layers a be neural nets with uh a lot of layers a lot of um complex structure to it um to

[33:32] be able to uh learn from data. So learning from data is machine learning, right? But using neural nets to do that is deep learning. So that's where we're at is using neural nets in particular, moving away from just basic machine

[33:49] learning into more specialization here with neural nets. So, um, what's nice about neural nets is that it can work with structured data, but as I've said, it can unlock the ability to work with unstructured data from many different

[34:04] domains such as imagery, audio, video, text um, to make predictions uh, usually text um, to make predictions uh, usually um, and the uh, this has unlocked a lot

[34:16] vision, so object detection, self-driving cars, those kind of things. NLP which is dealing with language so language tasks like sentiment analysis, translation, text generation, um even working with audio data like

[34:31] speech recognition. So a lot of progress in those fields and deep learning is going to be able to surpass a lot of the machine learning techniques that we've seen because they have a very uncanny

[34:44] seen because they have a very uncanny ability to learn complex patterns. Um, one of the things that is that gives that the ability is how complex the model is. But the other thing is it's going to use a lot of data. So it's

[34:57] going to have a lot of examples to go off of. It needs that in order to extract those complex patterns amongst imagery, text, audio, video, right? It's going to require a lot of data. So that's something we'll see when we start

[35:11] doing examples. Okay? So just to compare the two. So we're talking about deep remember deep learning really being a subset of machine learning. Machine learning is a little bit more broad um where we have you know unsupervised

[35:25] learning supervised. A lot of the problems in deep learning are still supervised. Meaning that there's still a label like if you have an image there's still a label of what that image is. Is it a cat? Is it a dog? Is it a giraffe?

[35:40] Right? Or one of those thousand categories like an imageet uh challenge. categories like an imageet uh challenge. So there's still a supervised nature to a lot of these deep learning problems. It's just the model that we're using is

[35:53] now a deep neural net, right? Instead of a decision tree, instead of a random forest. And also the data is inherently different, right? It's going to be imagery. It's going to be text potentially audio video. So, we're

[36:07] potentially audio video. So, we're moving away from structured data and really focusing on unstructured use cases with things like images and text. One of the things that's true that we've seen so far is especially with machine

[36:21] learning course we just finished, we saw that it's really important to get those features right. you know, we have to do some data prep to get our features correct, get rid of any null values, maybe even engineer some features like

[36:35] scaling it or adding two features together. Um, it's really important to do that. What we're going to see with deep learning is it's not that important to do that. It's not that important to do feature engineering at all because

[36:49] then the network's going to be so powerful at picking up on patterns from the original data like the original images. um we're not going to need to do a lot of manual feature engineering with with

[37:01] deep learning. So that's going to be one kind of advantage here is not a lot of manual uh feature engineering. And that's actually the the imageet uh

[37:14] challenge we just talked about with Alexet uh convolutional net winning that challenge. That was the thing everybody was doing was trying to do a lot of manual feature engineering so that they could use like a a classical machine

[37:26] could use like a a classical machine learning model like a XG boost but doing really good neural net that can just extract patterns from the raw data very effectively using something like convolution. [snorts] Okay. So deep

[37:40] learning as I've said really excels in those kind of tasks like image those kind of tasks like image recognition, NLP, speech uh recognition. Um so we've talked about that. Uh it NLP remember is short for natural language

[37:52] processing. So you'll probably hear me use that word NLP especially when we get to later on and we're studying things like transformers and RNN's. Those are just language tasks, right? So think of

[38:04] text working with text, text generation, sentiment analysis, uh speech or language translation I should say. Um so processing natural language which is really text right for for our use cases. Deep learning often

[38:20] for our use cases. Deep learning often requires really large data sets. So much more data than you see in a machine learning technique. Um and also more computational resources. So this is think GPU. Most deep learning is going

[38:34] to benefit from using a GPU. Whereas machine learning doesn't really use that at all. Like if you do a decision tree, an SVM, a logistic regression, you're not using a GPU at all. You don't need to. You it will still be pretty

[38:47] effective um without that. And uh whereas deep learning, a lot of models will take forever. Just to give you some context, like the um GPTs of the world, these transformers that are, you know, neural network based, they train on data

[39:03] that's about the size if you took all of the text from Wikipedia and more. So all the text from Wikipedia um plus maybe all the text that would be in an encyclopedia collection. Think of a data set that big as a

[39:19] prototypical training data set for something like a GPT and think of something like a GPT and think of thousands of GPUs working together. Even thousands of GPUs working together. Even in that case, it can still take days to

[39:32] [snorts] train a model. Days, not minutes, not hours, days. Because the models are so big and there's so much data being processed to train it. that

[39:44] data being processed to train it. that is how much time it takes to train those kind of models. So there's substantial amount of resources um which is why there's so many uh news

[39:56] articles written about you know Nvidia and partnering with these uh generative AI companies like OpenAI and Microsoft etc. So a lot of resources are are needed. Now for the networks we're going to build in the in the program not so

[40:11] much. We'll we'll still explore using a GPU and we're actually going to practice GPU and we're actually going to practice using a GPU. Um we will have access to free GPUs in collab. So that's really nice. Or even the simply learn lab

[40:23] environment has free access GPUs. Um so we will practice using it and it will benefit us for things like image recognition working with imagery or recognition working with imagery or RNN's or transformers even. Um they'll

[40:37] benefit us. But just as a rule of thumb, deep learning requires more data, more resources to to train computational resources. Well, so if you if you have a

[40:50] Windows machine, sure, um you can use the Nvidia GPUs. If you have a Mac, um, like a MacBook, um, even like one of the M1, M2, M3, M4 series, those have

[41:06] built-in GPU capabilities that, uh, we can take advantage of. So, I'll share those resources with us when we get to building our models. But even if you building our models. But even if you have like one of those M1, M2, M3, M4

[41:18] have like one of those M1, M2, M3, M4 Max with the the silicon chips, they can use their internal GPU to train models. Um, and I've done that before and it's really nice. It speeds up the training a lot. We can also use Collab. So Collab

[41:32] lot. We can also use Collab. So Collab offers free GPU access, uh, which is nice. Um, so we can use that too and we will. So we'll practice that, uh, as when we get to it. Okay. And then finally, deep learning. You know, we're

[41:45] going to be using neural nets that are potentially very deep in the sense that haven't studied what a layer is yet. That's okay. Just think of a really complex model with a lot going on with it. That's going to be, you know, way

[42:01] more complex than something like a decision tree or SVM or random forest decision tree or SVM or random forest even. here. to train deep learning, we uh often will use GPUs uh because they're

[42:15] going to be really effective. Most traditional machine learning methods that are not deep learning can get away with just using a CPU because they have very simple algorithms that are that are training that do not require using the

[42:29] um underlying GPU resource that can really speed up uh uh deep learning. really speed up uh uh deep learning. Okay. through some of the successes that have been recent, some of the foundational

[42:46] kind of moments. Um, and there's been growth in a lot of areas. So working with audio, computer vision, even reinforcement learning that has been powered by deep learning. Um, most reinforcement learning algorithms prior

[42:59] to deep learning were based on what are called kind of tabular methods. um where called kind of tabular methods. um where they were uh keeping track of um uh they were uh keeping track of um uh rewards and and uh data in in a more

[43:14] structured format. Uh but deep learning has since made a lot of progress in simplifying that and and uh making it much more powerful and and even in reinforcement learning. We'll talk about one of those uh cool moments there. Um

[43:29] talked about this in 2012. This was the champion in the renowned uh imageet challenge. This is where again they used convolutions to really power

[43:43] learning imagery and being able to classify images. And this this was so groundbreaking that uh a lot of moments in deep learning over the years have

[43:55] been referred to as kind of a imageet moment or Alexet moment um because how big of a breakthrough this really was in 2012. Um these this convolutional model was kind of created. Now we'll study this architecture later on of what this

[44:11] actually means, what these uh layers are doing, what is a convolution. Um we'll computer vision. But this was a huge breakthrough in working with image data.

[44:30] breakthrough of working with text known as uh the first kind of word embedding model which is really really important for where we are today because word embeddings play a really critical role in working with text. Uh because these

[44:43] in working with text. Uh because these are um models that map text into numerical vectors which is really critical for um working with language modeling. And every transformer really has a word embedding at the very front

[44:58] has a word embedding at the very front of it to take raw text or tokens and essentially um make those into a mathematical vector so the neural net can work with it. The very first uh model there that was a big breakthrough.

[45:11] This is 2013 um was a model called wordtovec uh which uh you may have heard of. This was uh used for uh pre-processing text was uh used for uh pre-processing text essentially to to map it from raw word

[45:25] essentially to to map it from raw word or tokens into uh vectors. So really really critical for any kind of language model to do this. And so this this has the language models but this is a really

[45:38] important breakthrough for them for for generative AI. So every generative AI generative AI. So every generative AI like GPT cloud every everybody has a word embedding model that's at the very front of their uh transformer-based u

[45:52] front of their uh transformer-based u text generation model. So this is this is a huge breakthrough. Then we have the very first kind of uh language models known as sequence to sequence. Now this is not using a

[46:06] sequence. Now this is not using a transformer. So this is using RNN's to transformer. So this is using RNN's to try to do NLP tasks like a language translation from uh German to English or French to English or or English to

[46:20] French to English or or English to French whatever it is. And they tried to French whatever it is. And they tried to make NLP so doing things like sentiment analysis, they tried to make it a lot better and it did. So, at the

[46:33] time, RNN's were kind of the best language models around, and this is language models around, and this is 2014, but they quickly get surpassed by And that's where we are today. Everything is kind of transformer-based,

[46:48] but at the time, RNNs were kind of the best uh for language. We will study RNN's uh later on in this course. recurrent neural nets, RNN's, which are just a flavor of deep learning that uses a a special architecture. So, here it is

[47:05] There's a lot of different variations on the RNN uh architecture um that attempts to predict um again what word should be next in the sequence using the RNN. We do that very well today with transformers rather than

[47:22] RNN's, but at the time, these were the best. uh around okay right around the best. uh around okay right around the same time in the image generation uh field we have GANs as a big breakthrough. So what what are GANs?

[47:36] They are uh generative adversarial networks. Um these are models that networks. Um these are models that generate images fake images. Um, and they do it through a competing uh adversarial networks that um, one is a

[47:53] generator that builds images, the other is a discriminator that's supposed to predict if that image is real or fake. And they kind of boost each other up so that the generator by the time the training is done is so effective at

[48:05] generating fake images, they kind of mirror real images. This actually led to the explosion in deep fakes. So, deep fakes became a real issue.

[48:17] There's still issues today, but the the first issue, the first time it was an issue is really when GANs were uh prominent. They were developed in 2014. prominent. They were developed in 2014. So, again, not too long ago. Most models

[48:31] that generate images today are actually not GANs. They're what are known as not GANs. They're what are known as diffusion. So like Dolly and in and the Google banana models that you can um put in some text and generate an image. Um

[48:46] they're not GANs but GANs were the earliest kind of image generation models that were out there and really effective and some of them are still used today for sure but majority of image generation models are are what is known

[48:58] generation models are are what is known as diffusion which we will study later on generative AI but this was a huge breakthrough in using neural nets to generate images right that idea of

[49:12] generating images through neural nets this was a big breakthrough here 2014. Okay. And then on the reinforcement learning front, um we have the Alph Go developed by Google and its Deep Mind team. First, uh kind of machine program

[49:27] to defeat a professional Go player. So, if you've never heard of it, Go is a board game that is thought to be more complex than chess cuz the board is a lot bigger. um it has more complex rules to it and more [clears throat]

[49:42] uh more states in terms of the the the board. There's a lot more variability in in the pieces uh in their positions on the board. So, it's thought to be a more complex game and it was mastered by

[49:58] reinforcement learning using deep learning. So this is the first time where deep learning really benefits a reinforcement learning model so much that it's able to beat uh go professional go players. Um this was a

[50:12] huge breakthrough 2016 about 10 years ago huge breakthrough especially in reinforcement learning and this is where deep you know deep learning being able to be used in RL big deal. It wasn't always that way. Okay now to the big one

[50:28] which is 2017. This is what I alluded to earlier with transformers. earlier with transformers. So this is where um the very first transformer was created. We're going to study transformers later on in the

[50:41] course, but transformers changed everything when it came to language everything when it came to language task. So I said RNN's were used prior to transformers around 2014 to do a lot of language tasks. Transformers come along

[50:54] a few years later and really revolutionize everything. And all of the revolutionize everything. And all of the generative models today like Chad GBT, like Gemini, like Claude, they're all based on Transformers. They are just

[51:10] massive extensions of the Transformer idea. Um, these large language models, these LLMs are all based on the transformer idea. This was a huge transformer idea. This was a huge breakthrough. I remember myself um I had

[51:25] just gotten into data science, machine learning in the industry right around learning in the industry right around here like 2016 and I remember reading this paper at the time. Uh and how important it was. It's just a huge

[51:37] breakthrough. Uh the attention all attention is all you need. Highly encourage you to go and read that if you can. It's it's amazing. Now it might not make sense yet. By the time we get to transformers, our lesson on transformers

[51:52] later on in this course probably make more sense. But um really fascinating breakthrough. This is 2017, not that long ago, right? Really not that long long ago, right? Really not that long ago. And not too long after this, we see

[52:05] the explosion in generative AI, right? Especially for text. Okay, so those are just to name a few. Obviously, there's been other advancements over the years, but those are some of the biggest. Um, and uh,

[52:21] are some of the biggest. Um, and uh, what we've seen is deep learning has really come a long way. There's so many cool algorithms. There's uh, there's also been an explosion in in implementation uh, um, because we have

[52:34] these um, frameworks readily available that are open source that anybody can use. So more it's it's more and more people have access to these libraries algorithm in the community. So the research community has benefited a lot

[52:51] and we've seen because of that an explosion in AI tools and uh AI models because of the ease of use of these deep learning um frameworks and and all the

[53:06] advancements that have come over the years. So a lot of cool uh libraries, a and we're going to study those like we're going to study TensorFlow, we're going to study PyTorch so we can build our own networks. But the community has

[53:19] benefited greatly and has been part of the reason there's been an explosion the reason there's been an explosion recently is because of how easy it is to get access to those libraries and start building your own neural nets and trying

[53:31] things out really easy. Okay. So that brings us to uh an important question which is why should we study deep learning? I said before that you know it's an advanced topic. Not everybody studies it. It's a bit of a

[53:46] specialization within machine learning and a lot of people just get through data science and machine learning and kind of stop there. So what reason do we have as learners to actually study deep learning? I'm going to talk about that.

[54:00] learning? I'm going to talk about that. One of the key reasons is that deep learning as I said before unlocks problems across many different domains that were that are difficult but are necessary in those domains. So I'll give

[54:13] you one example is healthcare. There's so many image related examples in healthcare that are really vital that neural nets help with significantly. So neural nets help with significantly. So think about analyzing images and image

[54:28] think about analyzing images and image data of the body or of um the brain of um potential diseases. Those can be more readily analyzed with deep learning. Um and so healthcare's seen a lot of advancements in analyzing

[54:44] that kind of data with with deep learning, autonomous vehicles of course, robotics, um even like fraud detection. There's a lot of advanced methodology there or even like time series forecasting has

[54:59] benefited from uh neural nets. So there's so many advanced use cases that neural nets can benefit um because it can work with text, because it can work

[55:11] with image data. Um it just unlocks a lot of possibilities. Um so so that's one reason is it just gives us many more problems we could potentially solve in in different domains. It's worth studying it there. Another thing is we

[55:27] have so much more access, right? So, um, GPUs are more readily available. Um, cloud environments have GPUs readily available. We're going to see that in Collab, like we are going to be able to work with GPUs in Collab for free. But

[55:42] certainly, like if you're paying for it on a on a cloud provider like GCP, AWS, on a on a cloud provider like GCP, AWS, Azure, they all have GPUs that are relatively cheap. Um, considering that we're actually using, you know, advanced

[55:57] we're actually using, you know, advanced hardware there. Um, so because of that, it is because it's so readily available, we have a better chance at utilizing deep learning more than ever because GPUs are so so much uh easy to get your

[56:13] hands on than ever before. So, it's not to say it's perfect. Like, you know, we to say it's perfect. Like, you know, we were joking earlier about the in Nvidia um 5090. you know, there are some physical hardware elements that are

[56:26] expensive and not easy to get your hands on, but like cloud resources for GPUs um are readily available pretty easily. We're going to see that with collab in We're going to see that with collab in particular. Another thing is uh you know

[56:40] GPUs I I forgot to mention this but let me go back to this slide actually is um one of the things that Nvidia really powered was this uh CUDA framework. This is something that I mentioned a little bit earlier. This allows the Python

[56:56] library to interact with the GPU and be able to offload operations to it instead of having to keep everything on the CPU which is much slower for the neural net operations that that it's doing and its layers. Um and so in uh there was kind

[57:12] of this um framework to interact Python with the hardware this CUDA um library with the hardware this CUDA um library and um because of that the libraries that manage neural nets like PyTorch and TensorFlow have integrated the CUDA

[57:27] TensorFlow have integrated the CUDA framework and have the ability to easily framework and have the ability to easily manipulate operations on the GPU um which makes working with uh neural net so much easier. So you can take

[57:40] advantage of a GPU. That's just a long way of saying you can take advantage of GPU really really easy with the modern framework. So with PyTorch, with TensorFlow, they can interact with the GPU so easily because of things like the

[57:54] CUDA framework um makes it so much easier. So the the CUDA is we don't ever see it. It's abstracted away with the framework like PyTorch or TensorFlow. We're going to see that we are going to build models in

[58:07] PyTorch and TensorFlow that will take advantage of the GPU. So, we're going to see this directly how easy it is to truly take advantage of that hardware and it's going to speed up our operations a lot. It's it's going to

[58:20] make it so much faster to to do that processing. Okay, so we just said GPUs uh play a critical role um and they're more readily available than ever. So that's another reason to kind of learn deep learning is we have access to these

[58:34] deep learning is we have access to these resources and it's it's really because the GPU can do parallel math operations so efficiently way more than a CPU. So

[58:46] they they really we're going to see the effects of this you know when we do especially working with um computer vision and convolutional networks working with a GPU then becomes almost necessary. It'll speed up operations so

[58:59] much like if we didn't use it, we'd be sitting there for minutes, maybe hour waiting for the model to train, but if we use a GPU, it'll take seconds. It'll really speed it up a lot. Okay, one of the other reasons we we've kind of

[59:12] hinted at this is just a low barrier to entry. So, so this is just to say why machine learning. There's so many problems that use neural networks and

[59:25] why not learn deep learning because it's never been easier to do. There's hardware available with GPUs more than ever. We talked about that. There's user really extremely userfriendly frameworks like TensorFlow and PyTorch to help us

[59:40] build neural networks and they're open source. So they're easily we can pip install it, get it into our environment and start building neural nets and training them. really really easy and we already have all the fundamentals we

[59:53] need to really understand model building, model training, um model evaluation. We have all that fundamentals from working with uh Python, working with uh machine learning that we've already done, data science.

[1:00:08] So it's pretty easy to get into at this point. So why not, right? it it's just point. So why not, right? it it's just going to help us uh expand our the breadth of our skills and problems that we can solve. And like I said, open

[1:00:22] sourcing these like TensorFlow and PyTorch was a big deal. It made it so much more accessible for the community to build their own neural nets. Prior to that, it was really limited to these big companies. You had to be in one of them

[1:00:35] in order to really start building neural nets. But now anybody can because these are free and open source like TensorFlow and PyTorch. Okay. So another key reason and you may be wondering this especially because of

[1:00:49] be wondering this especially because of generative AI um taking over uh a lot of development these days but still there's a need for trained deep learning experts a need for trained deep learning experts and practitioners. So many um companies

[1:01:03] still work with deep learning models and need deep learning expertise um to be able to like tweak the training, set up the training, set up the model, um make the adjustments if the training doesn't go well, really

[1:01:19] understand the model. Even if you're using generative AI, you still need to modeling works and how it [clears throat] functions in order to [clears throat] functions in order to effectively use it. So there's still a

[1:01:31] need for deep learning expertise regardless of AI and generative AI. So it's a good opportunity. It's a it's a good skill to have. Um I can say this like beyond just having data science and machine

[1:01:47] learning experience going through this program, I think having the deep learning experience is just another really um beneficial thing to have if you're trying to get into the industry. it it really unlocks more potential for

[1:02:02] you I think and again not everybody studies it so you're getting a kind of advancement above the competition potentially by learning deep learning all right let's talk about some applications so there's many as I said

[1:02:16] before there's many different uh fields that really benefit from uh deep learning and working with neural nets um so here's some of them just a few you know um NLP being a huge huge one in terms of processing text, being able to

[1:02:32] understand human language. That's a big one. Uh self-driving cars, using computer vision for object detection. That's something we'll talk about and study. Um not only like detection of objects, but classification of them. So,

[1:02:46] objects, but classification of them. So, think of like, okay, here is an image and inside of it, there's a pedestrian over here. There's a there's a a a sign like a a stop sign. There's a tree over here. There's another car over here. Um

[1:03:03] here. There's another car over here. Um here's the road um divider. Um so so many objects that are that you can detect in the environment and really classify what they are. And that's all powered by deep learning. Same thing

[1:03:15] with audio processing, transforming speech to text. That's how we talked about earlier. That's how things like Alexa and those voice assistants really work. Translation of audio into text. And then uh manipulating images as well.

[1:03:31] So generative AI with images. So much of generative AI is all deep learning based transformers um diffusion models that uh generate images, GANs, all of these generate and manipulate images. They're all based on

[1:03:46] deep learning. So much of generative AI is deep learning based. Okay. Now there are some limitations. Uh one of the things to realize about deep learning, we've talked about it a lot so far is a significant amount of data is usually

[1:04:00] required. Meaning that deep learning may not be the best for tasks that have um little amounts of data. And so what is little? Um usually we're going to be

[1:04:12] little? Um usually we're going to be dealing with uh in the tens of thousands if not more. That's like a that's on the very low end of maybe images or text examples in order to train a moderatelysized neural net. Um maybe

[1:04:27] even on the smaller end of a neural net. The bigger the neural net that we have, the more complex of a model we have, the more data we need. Um, so this is this is like for the problems we've dealt with so far, throwing a neural net at it

[1:04:44] would likely be overkill and would not be effective. So it's not like we can just throw a neural net at our housing data and expect it to predict the price very effectively. It doesn't work like that. We would need a lot more data for

[1:04:59] that. We would need a lot more data for that to be effective. Um, and we would probably need to tweak the neural net. uh a good amount to get an effective model there. The neural neural nets have a tendency to overfit. So they they need

[1:05:14] a lot of data to avoid that overfitting. They need a lot of examples. Um and and that can be sometimes difficult to come by. Um the amount of data that's required to be collected. Generative AI like GBT requires so much data. That's

[1:05:30] why they're using people's chats to continuously build a data set so that it continuously build a data set so that it can keep training it. Um so much data is required there to train a really effective model especially one that big.

[1:05:44] effective model especially one that big. Um just requires a huge data set. Um let me give you an example. So consider you know a model that's we've just differentiating or classifying cats and dogs. So this image is a cat, this

[1:05:58] and dogs. So this image is a cat, this image is a dog and etc. Um, in order to image is a dog and etc. Um, in order to to do this effectively, we would need many different images of cats and dogs and and likely those images would have

[1:06:12] to be of different size cats and dogs, different breeds, um, from different angles potentially from different lighting and brightness of the image. um many different variations on those and they'd also have

[1:06:26] to be labeled, right? So it's it's still a supervised learning problem. Um so you know gathering that data set may be an expensive endeavor may not be trivial. expensive endeavor may not be trivial. So that is kind of a a cost burden of

[1:06:42] doing deep learning is the data. It's going to be a lot. um you may need a really massive data set and that can be difficult to acquire in some instances, not all, but in some cases that may be difficult to acquire. So, it's just a it

[1:06:57] is not a limitation so much as it is a caution that if you're going to use deep learning, you generally need a lot of data and so you have to be prepared to collect a lot and process a lot in order to um have an effective model. The other

[1:07:12] thing is hardware. We've talked about this now. it it's again not necessarily this now. it it's again not necessarily a huge limitation because we do have access to GPUs but you may need to pay for it um and you may need advanced

[1:07:27] hardware the more complex your model is the more you're going to benefit from you know uh a more complex hardware like a GPU um it is more readily available than ever but it doesn't mean that it's always going to be free and always going

[1:07:43] always going to be free and always going to be easy to uh acquire. So there may to be easy to uh acquire. So there may be some work there to use that or to acquire that hardware um for a really complex model. Now that being said, we

[1:07:57] are going to work with free GPUs for the examples we will do, but the examples we will do will be relatively moderately complex examples. they won't be massive [clears throat] models that would require a much more significant hardware

[1:08:12] investment to to be able to run. So I used the example before um something like a Jad GBT is going to be using thousands of GPUs. That's a significant investment. If you're just using one for free from like Collab to do simple

[1:08:26] examples, that's one thing. But for a really big network, um, you're going to want to use a significant amount of GPU resources, um, which may be costly,

[1:08:40] right? So, it's just something to be aware of. mentioned earlier is that deep learning because there's um, it's such a complex

[1:08:53] because there's um, it's such a complex neural network, it can be susceptible to overfitting. Um, in fact, they are very susceptible to overfitting, can easily overfit, especially if there's not a lot of training data. And that's just the

[1:09:07] nature of the neural net. It it's so complex of a model that it can effectively memorize the training data very easily if you don't have guardrails we're going to study in that lesson on optimization of the training process.

[1:09:23] certainly things we can do and we're going to study to prevent a neural net from overfitting. But they are very prone to it. Naturally very prone to it because of their complexity. Right? Remember remember we talked about

[1:09:38] overfitting. The more complex a model is the more likely it is to overfit. The more basic a model is the more likely it is to underfit. So neural nets fall in that category of being really complex. So they are very prone to overfitting.

[1:09:54] So something we have to kind of uh be on the lookout for. We'll study some techniques to help us uh overcome overfitting or prevent it. All right. Another one that is very underrated and a lot of people don't think about when

[1:10:09] it comes to deep learning is the explanability of deep learning is very very limited. So when we studied machine learning and we did things like logistic learning and we did things like logistic regression or a decision tree that was

[1:10:24] super explainable because we like in logistic regression you get coefficients on every feature right you get those on every feature right you get those betas. So we know exactly which features

[1:10:37] are important because they're going to have higher coefficients, right? And same thing with like a decision tree, you get an actual tree structure so you know that you know the path that leads to a prediction. And so they're really

[1:10:50] readily explainable. The issue with neural networks is they're often a black box. And and what that means is you put data into it, you get a result out, but

[1:11:02] it's almost impossible to explain every little thing that happened that led to that prediction. Very difficult to explain with neural nets because they're such a complex model. They don't have a very natural way of um having like a

[1:11:18] single coefficient. Most um to to give you some context, most modern neural you some context, most modern neural nets like like a uh GPT that's nets like like a uh GPT that's generating text is going to have in the

[1:11:32] billions, tens of billions, hundreds of billions, if not trillion weights, uh which are, you know, think of those betas. It's going to have like of those betas. It's going to have like a trillion of those betas. So it's

[1:11:46] impossible to say what a single contribution to the prediction is. Now this is an active area of research in the research community is people are the research community is people are trying to figure out good ways to um

[1:12:01] break down how a neural net's making a prediction. But it still is near is pretty much not possible relative to other machine learning techniques. models. You put something in, you get something out. It's hard to say what

[1:12:15] happens in between. You know the architecture, but it's hard to say what exactly contributes to that final result. It's hard to hard to trace that result. It's hard to hard to trace that back. Um, so uh deep learning can be

[1:12:29] very difficult for explanability and and this actually has an effect in like regulated environments. Let's say you're in the finance industry and you're making um like loan decisions based on a model. Um you probably don't want to use

[1:12:44] a a neural net because it's going to be hard to explain how you predicted your like uh loan or not loan decision. You you probably want to use something that is more explainable like logistic regression or a decision tree which

[1:12:57] would those you can directly see the contribution of their features, right? neural net you're not going to be able to see that. It's just too complex of a model. Okay, so that's another kind of drawback of deep learning. Now there on

[1:13:13] that we don't care about the explanability at all. So you think about explanability at all. So you think about like a self-driving car. We don't care the explanation of why this is a stop sign, why this is a pedestrian, why this

[1:13:27] really fast. It needs to be real time. It needs to be really fast. we don't care about an explanation in that case, right? We just need it to make an accur accurate prediction. So there are certainly cases where we don't care

[1:13:39] about that. But in a lot of like regulated industries, you may care about that. Uh if you're if you're using a model to make some type of decision um you may care about the explanability of it in which case neural nets are not

[1:13:54] ideal for that. Okay, let's talk about some of the uh the frameworks then that some of the uh the frameworks then that we will use. So these are going to be uh basically our libraries and tools to um

[1:14:08] basically our libraries and tools to um help us develop neural networks. Um help us develop neural networks. Um basically they um they allow us to design neural networks. They allow us to train them. They allow us to debug them

[1:14:21] like how they're working or what kind of uh evaluation we're doing if they're uh evaluation we're doing if they're producing any uh any bad results. Um, and also for deployment. Now, we won't really focus on deployment uh too much.

[1:14:35] scope for us a bit. We're going to focus on um how to design and kind of train them. But these Python frameworks will allow us to build and train and evaluate

[1:14:50] uh neural network neural networks. And this is huge like like I said earlier, it wasn't always this case, right? you really had to be part of these large corporations that were doing this like a Facebook or a Google in order to have

[1:15:06] access to the tools to build neural networks and now they're readily networks and now they're readily available like PyTorch and TensorFlow. So the main the main uh frameworks that we're going to study are uh PyTorch and

[1:15:20] TensorFlow. Now caris is on this screen because what we're going to learn is because what we're going to learn is caris is really um embedded into tensorflow. It was actually developed by Google the same people who made

[1:15:33] tensorflow. It's really just a part of it. It's it's an interface to core tensorflow. So when we hear the word caris we should really be thinking tensorflow. It's the same thing. Um it's actually technically built on top of

[1:15:48] TensorFlow and allows us to interface with TensorFlow but so so these these were built by Google and then um PyTorch was built by

[1:16:00] and then um PyTorch was built by Facebook and now Meta um but at the time Facebook and now Meta um but at the time Facebook um and they are open source now so they're widely available and of of course we will learn how to use them to

[1:16:12] course we will learn how to use them to build neural nets and uh uh train them. Um they're both widely used. So both widely used community. Most people ha or the other just based on either what they originally learned to begin with or

[1:16:30] um what they just naturally like. But the the truth is there's uh a lot of models out there like even like an open-source GPT that exists or Llama,

[1:16:43] they're developed in both frameworks. They have a version that's developed in PyTorch. They have a version that's developed in TensorFlow. And it's they do the same things. They mostly just have syntax differences. That's the only

[1:16:57] only thing about it is they primarily just have syntax differences. But again, it's worth our time to study both because both are so widely used. Um, and

[1:17:10] honestly, both are still maintained. Even though they're open source, both of these companies still maintain uh groups of developers that are maintaining these libraries. So there's still people

[1:17:23] within Google who actively manage TensorFlow and there's still a group within within Meta now that's actively managing PyTorch and they they do work source so contributions can come from anywhere really but there's dedicated

[1:17:40] kind of people within each of these uh companies that are still maintaining these frameworks because they're so critical right they're so critical to doing deep learning um and doing neural networks. [snorts] So we're going to

[1:17:54] study them uh pretty deeply. So caris let me talk about caris. It is um going let me talk about caris. It is um going to be a a interface in python to the tensorflow library. So tensorflow itself. So tensorflow was created first

[1:18:09] and then caris came along a couple years later um to interact with tensorflow. TensorFlow is kind of the underlying library that Caris is built on top of

[1:18:21] and it just it's much much much more user friendly to manipulate and manage neural networks. So most people gravitate towards using the caris library within TensorFlow um to building if they're going to use

[1:18:36] TensorFlow, they're going to be using caris as as the interface to TensorFlow. So it's just built on top of that by the same Google group who made TensorFlow. And it's it's the advantage of it rather than using the original TensorFlow is so

[1:18:51] build neural nets and so much easier to manage them. It's a really nice interface of TensorFlow. So when we build models in TensorFlow, we're really going to be using caris, okay, as that um kind of library on top of uh

[1:19:07] TensorFlow um to to build to build neural nets. So we will study that. Is there an advantage of using caris directly? Yeah, that's what it is. It's it's a really userfriendly interface. So it's

[1:19:23] designed to make uh building TensorFlow models so much easier because the truth of it is like core the original TensorFlow is kind of ugly. It's kind of

[1:19:35] TensorFlow is kind of ugly. It's kind of ugly and like messy to deal with. So um people when they're building TensorFlow models gravitate towards using caris as the interface to tensorflow because to like core TensorFlow is really messy.

[1:19:50] like core TensorFlow is really messy. Um, TensorFlow is, as we'll learn later, is is basically a graph library for building like a tensor graph to to uh building like a tensor graph to to uh mimic a neural network, a graph of uh

[1:20:04] computation. But Caris makes it so much easier to to work with TensorFlow. So most people gravitate towards caris then just you usually you won't ever use the original TensorFlow unless you're maybe doing specific research that would

[1:20:21] require it. You're mostly always going to use caris to build your models and train them. So that's what we'll do. Okay. So underlying caris is TensorFlow which came first um from Google. uh it's you know obviously within Python it's

[1:20:35] open source now and this was the original library they built in-house to to manipulate and manage neural networks you know caris came along as the interface to tensorflow to make working with tensorflow so much easier um so

[1:20:53] actually tensorflow has some really cool tools uh uh as part of it that we will uh work with later on they have tools for like visualizing ing the training process which is really nice. Um so we'll work with those.

[1:21:09] Okay. So on the other end of it is PyTorch that uh that Facebook developed now Meta um who still maintains it but it you know has been open sourced. um

[1:21:22] same kind of thing they were trying to work with neural networks and needed an work with neural networks and needed an in-house you know in-house library to do that and so they developed PyTorch at the time you know and since it has been

[1:21:35] open source to the community and it's it's very similar to TensorFlow honestly has the same kind of features it as TensorFlow um it's just a different syntax and uh so that we're going to work with both as

[1:21:51] I that we're going to work with PyTorch and with TensorFlow um in order to build and with TensorFlow um in order to build models and and uh train them. So, as I said in the community, it's kind of a 50/50 split on which one people have a

[1:22:05] 50/50 split on which one people have a preference for. I personally um like PyTorch. I I I actually learned TensorFlow first um but found myself liking PyTorch a little bit more, but it's preference. Honestly, it's

[1:22:20] preference and I've seen um it it really just depends on which what you like better. And then oftentimes like where you're working also determines, you know, maybe everybody has a standard for okay, we're all going to use TensorFlow

[1:22:34] or oh, we're all going to use PyTorch. Um you see things like that. But honestly, if you know one of them, you will be well verssed in the other and and we are going to work with both. So we'll get exposure to both. just syntax

[1:22:49] differences really between the two. Okay, so what I wanted to do was wrap up lot in the introduction, but I wanted to wrap up specifically with um the life

[1:23:01] cycle of a deep learning project, which I think we're going to see mostly overlaps with the same steps we were doing for machine learning. It's just there's going to be more, you know, there's going to be more details that

[1:23:15] are that are involved here when we do training, when we do uh model building, training, when we do uh model building, but mostly the life cycle this is learning from a high level, right? It's mostly the same. We have some type of

[1:23:30] data collection and data prep phase, which is kind of what this is. So, so this is like our data prep that we do for machine learning. We're used to for machine learning. We're used to doing that um first usually. Now, prior

[1:23:44] to that, there may be some planning that has to be done uh to collect data and and in deep learning that may actually be a significant amount of work to um go

[1:23:56] out and gather a bunch of images or process a bunch of images or text data, you know, and label it appropriately. that may actually be non-trivial but uh you know all the examples we will do will assume we already have that data

[1:24:11] but other than that it's mostly the same steps in term at a high level of like once we have our data prepped we go ahead and and um train we build and train our model um obviously the details of doing this with a neural net are

[1:24:26] basic model like logistic regression or something um but once we train the model we then evaluate right? We evaluate it on some test data,

[1:24:38] evaluate and see how it performs. And that and the reason there's an arrow backwards is based on that evaluation, we may need to make make some tweaks. So, you know, at each one of these, it could be very iterative. Um, in the

[1:24:53] training, like if we're not if we're underfitting, we may need to go back and revise our data. We may need to revise our model even. Um, so it can certainly be iterative and then after you have a model you're very satisfied with in the

[1:25:08] evaluation phase, that's where you can get into kind of um deployment of it and then like monitoring it and m maintaining it. These two things we're

[1:25:20] because again it's going to be a little bit out of scope for us. um we're mostly going to focus our attention on these two phases training and evaluating of

[1:25:33] neural nets. Okay, but let's talk a little bit more about these steps. So little bit more about these steps. So the planning uh as I said this is where the planning uh as I said this is where um primarily whenever you're doing deep

[1:25:45] learning this third bullet is is really critical. So I would lean on this bullet as really critical as so planning really meaning what resource we going to need meaning what resource we going to need in terms of data what kind of data are

[1:25:59] we going to need like images text you know what are we working with here not only data but hardware like can we get access to like are we going to need a access to like are we going to need a GPU um can we get access to that for

[1:26:13] going to be really critical to know your resources ahead of time um and be ble to have those available especially the hardware um like a GPU. So usually there's some thought put into that from a planning perspective

[1:26:28] um in terms of the resources. Uh then there's the actual collection. So depending on the problem this could be very different right we may be physically taking a bunch of images we may be downloading a bunch of images or

[1:26:42] or grabbing them from some database. Um, we may have IoT sensors that we're collecting sensor data from or like our self-driving car is taking a bunch of self-driving car is taking a bunch of imagery sensor data. Um, so that could

[1:26:57] imagery sensor data. Um, so that could be different. Um, audio, you know, audio or video, we could be collecting for examples. We need to collect that and we also need to label it, right? We need to have adequate labels and like think of

[1:27:12] the cat and dog images, right? we need to take every single image and provide a label with it. Um, if it's a cat or it's a dog and we also need to gather those images, right, in the first place. So, there's going to be data prep that goes

[1:27:25] there's going to be data prep that goes on here, no different than before. training phase. So one of the things about the training phase is that's not really shown here but is definitely a prerequisite is we first need to build

[1:27:41] obviously we need to build I would say build slasharchitect architect uh the model right the neural net

[1:27:56] neural net so obviously we need to do that and that will feed into the rest of the training process. Now, I um we'll talk about this later. I'm not too just the steps of the training that we're going to learn. Um so, this this

[1:28:13] will really go backward and iterate through these steps. But training of a neural net is very unique. It basically involves sending a bunch of data through the network, figuring out what your error is, and then going back and

[1:28:27] updating the network. That's essentially what all these steps do. And then you and over and over again until your network uh learns how to effectively make predictions. Um so it's a very iterative process which is what makes it

[1:28:43] take a long time. You're going to iterate through all of your data and you're going to do that on a number of steps which could be thousands of times that you're doing that. And it's really the same thing over and over again of

[1:28:55] again sending data through computing how far off you are in the error using that amount of error to go back and update all your your model. Um this process is where that gradient descent is going to come into play. So we kind of touched on

[1:29:11] this in the recommendations with that matrix factorization learning. Um we are matrix factorization learning. Um we are going to study this in detail quite a quite a bit more here in neural nets because that's really what's going to

[1:29:26] because that's really what's going to power neural net training is um gradient descent. So again these things I don't expect us to know right now. Just giving you a preview of the training is a lot more involved. There's there's a

[1:29:40] significant amount of computation involved here in like sending data through, generating predictions, computing error, updating all of our weights in our network, which could be billions and billions and billions of

[1:29:53] weights. So this this back propagation step could be significantly computationally intense um in the gradient descent and you're doing this many many many times in iteration uh across all your data. So it is

[1:30:10] significant the amount of resources that are spent in training a neural net. Okay. All right. So there's that's the training and again we're gonna have a lot more to say about these in future lessons. So let's put these terms on

[1:30:23] hold until we learn more about neural nets. I'll we'll cover what forward propagation means, what loss means, what back propagation means. We're going to study all of those. So a lot more to come on those. But for now, just think

[1:30:37] of the training phase as we have to iterate through having our network learn adjust over and over and over again until it learns effectively. Okay. Um so

[1:30:50] as I said the model is goes through iterative uh performance um evaluation essentially in terms of how much error it's generating. And so we keep an eye on that and continue to train it over the course of the training phase which

[1:31:05] may be lots of iterations. And you can go backwards at this point. So based on the training, you may go back and adjust your model. You may go back and adjust your uh parameter like your hyperparameters of the training which is

[1:31:19] going to be a few. You may go back and adjust your data which is what this slide's saying is that we may have to go back and collect more data. we may have to double check our labels. Um there could be a lot of reasons that we need

[1:31:34] to go back and make adjustments. It could be we changed the architecture as well like we actually changed the model itself, the neural net. Um so going back to this diagram, there's a lot of iteration that happens.

[1:31:50] So you know based on the training we may go back and alter our data we may alter our model based on the evaluation we may go back and um alter our training hyperparameters a lot that we you know may be modifying

[1:32:04] during the course of the training. So the training is going to be more intense I think in neural nets than what we've seen to this point with usually what we've seen to this point is we just run a fit and everything happens for us

[1:32:18] right especially with scikitlearn right we just run fit and boom we have a model that's trained like a logistic regression or uh decision tree right dot fit and it's done this is not going to be as simple there's going to be a

[1:32:32] little there's going to be more setup more moving parts involved um more computationally intense for sure to do to do the training. Okay. But either way we slice it, there's still going to be an evaluation phase, meaning

[1:32:47] that you know even if we train, we still need to compute metrics to see if our model has performed well enough on the training data and on this testing data. So from a supervised learning perspective, that step will basically be

[1:33:01] the same as what we've seen before. Once we've trained the model, we're going to apply it to a hold out test set of data and decide whether or not the model's performing well. Is it overfitting? Is it underfitting? We can tell that on a

[1:33:16] test data just like we did with machine learning. That will not change. So what we're going to do is still have training data and a chunk of data that's probably smaller but still test data that we will

[1:33:32] train the neural net on this and then apply it on this to evaluate. Right? So that in machine learning. We're still going to do that here in deep learning. Um and the idea there is to make any adjustments

[1:33:47] performance we're seeing. This is what we just said. So still an evaluation Obviously just a different model this time. It's a neural net. Okay. Then if

[1:33:59] deploy it. And there's many different ways to do that. Not going to get into them in this course. But uh suffice to say that once we have done the evaluation, it's kind of ready to go and we would do a lot of um kind of

[1:34:15] packaging up and infrastructure and things to kind of host the model um many different ways to do that. But the goal here is to be able to actually use it to make predictions in in some some way, right? So maybe we're using it in like

[1:34:32] an object detection inside of a self-driving car or something. And of there, you're going to keep monitoring it and maybe make adjustments as you see the predictions coming in. So one of the goals that of even building the model in

[1:34:45] the first place is of course we're going to apply that to new data that has never been seen before and that's where things can go off the rails. So you really got to monitor the performance as you are predicting on new data, right? and see

[1:34:58] predicting on new data, right? and see how it's performing. So, um you may want to retrain it, make adjustments, maybe in the extreme maybe you have to go back in the extreme maybe you have to go back and uh change the model up. Um but you

[1:35:11] should be keeping an eye on the performance if you are using that model performance if you are using that model on new data. Okay, so uh just to wrap up here on this introduction lesson. So that's everything we've covered so far

[1:35:24] is just background information about neural net. We have not talked about what a neural net is technically from like a math perspective. We haven't like a math perspective. We haven't covered um you know how we train it. Um

[1:35:37] that's all to come. This is just background, right? So we've talked about deep learning being a specialized subset of machine learning that particularly focuses on using neural networks. We've talked about that and

[1:35:51] suited for unstructured data like images, text, audio, video. Um, so it's really really good at that. However, it requires usually a lot of resources to

[1:36:03] function. So it requires a lot of data, a lot of computational power like a GPU to be effective. And then in terms of the life cycle, mostly follows the same steps as machine learning. It's just those steps are going to be different

[1:36:16] because we're dealing with a different kind of model, right? and neural net. start lesson three. So, if you're following along, you want to go to those uh notes [snorts] and go to lesson three. So, we're going to start our

[1:36:31] study of neural nets and particularly with the most basic kind starting with a perceptron really like a simple model of a neuron and the idea is to build up from there into uh bigger and bigger

[1:36:45] neural nets. So what we're hoping to learn here is basically get into more of the details of how a neural net functions. So structure, functionality, um including starting with a very basic perceptron and then expanding that a

[1:37:01] little bit into multi-layer perceptrons and then talk about some of the um advancements beyond there that are out there like a deep neural net that that extends that things like CNN's RNN's we'll mention as well and then talk

[1:37:17] about some of the activation functions that are uh present in neural nets. how it it won't make sense this moment when I say activation functions but they'll make more sense when we get into uh what's involved in a typical neural

[1:37:30] neural net especially modeling a neuron um there's things like activation functions that are really critical we'll study some of the most popular ones um study some of the most popular ones um like relu sigmoid softmax um and then

[1:37:43] we'll talk about some of the uh some of the issues that can pop up during training in particular um things like vanishing exploding gradients vanishing exploding gradients Okay, so a lot to cover. Um, let's start

[1:37:56] with modeling the neuron, right? So, let's start there. Um, so obviously the let's start there. Um, so obviously the models that we're going to build are networks of neurons and we have to start with kind of the most basic model of a

[1:38:12] neuron and it's very much inspired by the biological neuron that is uh within the biological neuron that is uh within the brain. And uh so not that we need to know all of these things about it from a biology perspective, but just to know

[1:38:26] that the model of the neuron is obviously based on how a biological neuron kind of functions. Um so from this perspective you know there's inputs

[1:38:38] that are sent through they go through the various um nerve cells and then out comes some type of output signal that flows through to the rest of the network. That's going to be very much how the neural networks will function is

[1:38:54] there'll be kind of input data flow through this collection of neurons and then out on the other end comes this output data. So we'll see that as we go. So in a biological neuron there's these various components like the cell synapse

[1:39:09] being a connection between two cells axons transmitting the output dendrites receiving input. We're going to have equivalent components in the artificial equivalent components in the artificial neuron model that we are going to take a

[1:39:24] look at such as the perceptron. So in the artificial neuron we have a similar setup. Now this is really really critical to understand is that the

[1:39:37] artificial neuron is very much similar to a biological neuron in which case we to a biological neuron in which case we take all of the inputs and combine them together to generate some type of output. Now the basic model of a neuron

[1:39:51] output. Now the basic model of a neuron is going to do this really important is going to do this really important process of taking the inputs to the process of taking the inputs to the neuron. So those um could be let's call

[1:40:04] neuron. So those um could be let's call them x1, x2 and x3 or more generically however many there are xn and then weighting those with different weights. Okay, waiting those with different weights and then basically taking a

[1:40:19] weighted sum. So we have something like w1 w1 uh x1 plus w2 x2 plus blah blah blah plus w n xn. Now what does this remind us of? Hopefully

[1:40:34] this reminds us of something like a linear regression, right? It's very linear regression, right? It's very similar to that that that kind of does um remind us of a linear regression. The difference is that the neuron is

[1:40:48] actually going to process this weighted sum through what's called an activation function. So every neuron is going to be activated or deactivated based on th

[1:41:03] this flowing through some activation function. function. So you take that linear sum, you pass it through an activation and often this activation. So this f

[1:41:16] F is called an activation function which gets its name from the biological neuron of you know a neuron activating

[1:41:28] or not like or some people call that firing right a neuron fires um it activates. So this this f is called an activation function and essentially what the activation function is doing mathematically is applying some

[1:41:44] nonlinear function. So it's typically the activation is going to be nonlinear in nature. It's going to be some type of function that is nonlinear. Okay. and

[1:41:56] function that is nonlinear. Okay. and it's going to um it's going to be applied to that weighted sum of inputs to that neuron and then produce some sort of output which is the activation value. Um it's you know producing some

[1:42:12] value. Um it's you know producing some type of output. So in this case the neuron model this is the this is the model of the neuron is really just applying a activation to some weighted sum of inputs flowing into this neuron.

[1:42:28] Um so in this way it's kind of like the biological neuron where we had input signals flowing input signals flowing in and then some type of output signal coming out. Right? it it's basically

[1:42:41] coming out. Right? it it's basically that and then what the neuron is doing is doing an a weighted sum through an activation. That's all that the neuron is doing is doing the weighted sum summation of weights times inputs.

[1:42:58] So you're have you have your signals X you generate an output which is um the weighted sum through the activation and you get this output signal uh Y here

[1:43:10] training the model with lots of data determines yes exactly so during the determines yes exactly so during the course of um doing the computations the neuron will will be active or not active um it it's really not a matter of binary

[1:43:24] like active or not active it's more of what is this value that comes out of this activation function. That's the output, right? What is that value that comes out of the activation? This is this is this value here. It's that

[1:43:38] weighted sum of inputs through an activation is really the model of the activation is really the model of the neuron, right? So, what is that output is really what we care about. The reason we care about that is because I want you

[1:43:50] to think about where this is headed in a in a network. So in a network of neurons, what we're going to have is a collection of these guys. So we're going to have many more neurons that would fill up a layer of neurons. So this is

[1:44:07] just one. So we'll have a lot of these guys. And each one of these neurons could produce a collection of outputs, right? They could produce a collection of outputs um from their layer. And so and then

[1:44:24] um from their layer. And so and then what you what happens is these outputs become the input to the next layer. They the signals keep flowing, right? They go the signals keep flowing, right? They go into the next set of neurons,

[1:44:39] right? And then those um it what actually happens is you start to build this network. So not only does this go like this, but there's many more connections. So like this will become an input into this. This will be an input

[1:44:53] into this. This will be an input into this. And this will be an input into this. And same thing over here. Like this will be an input into this. This will be an input into this. This will be an input into this. And this would be an

[1:45:05] input into this. So everything from the previous layer which is coming out of previous layer which is coming out of that neuron all those neurons will flow forward into the next layer of neurons and on and on and on. however many

[1:45:20] layers you might have. So you can have a significant amount of these layers of significant amount of these layers of neurons in a really deep network. You might have lots and lots and lots of layers. So we're going to study that.

[1:45:32] We're going to stud study that as we go along. But that's really what we're building up to. Yeah, that's that's what we're building up to in a neural network. It's really laid out in this structure of neurons sending data

[1:45:45] forward into the next layer of neurons and on and on and on until you eventually reach a final output. Now I want to give you a basic use case of this neuron. I want to give you a basic use case of this output because it may

[1:46:02] be um it may be a little vague on what that output is. Um most of the time for that output is. Um most of the time for a neural network this output will generally be a probability. So this will generally be some type of

[1:46:17] probability especially for a classification. Probability will usually be a probability. So especially for a classification. So let's say um let's say we were

[1:46:32] predicting uh if it if an image like with with a brain scan is cancer or not cancer this would be like a probability of this would be like a probability of

[1:46:44] cancer that we generate like in a binary case right or spam or fraud right generally represents some type of probability it doesn't have to like if problem like let's say we were predicting the price for our housing

[1:47:00] predicting the price for our housing price data. This this could be a price, right? This could be like a full value uh some some floatingoint number price, right? That that uh that could be possible but it's a single output from

[1:47:14] this neuron that could represent um typically for classifications be a probability but could be a price and like in a regression problem. Okay, so again just to recap this because I think this is really really critical to

[1:47:27] understand moving forward in neural nets is what does a neuron do in the network? is what does a neuron do in the network? Every neuron is processing a weighted sum a weighted sum

[1:47:42] of inputs w2x2 w2x2 wn xn. It's a weighted sum of inputs through an activation function f. That's what every neuron is doing. And that

[1:47:57] produces this output value from the neuron, which is that activated value, neuron, which is that activated value, right? Some nonlinear function. And by the way, when we train the neural net, what we're actually learning during the

[1:48:12] training process, what gets updated during the training against real data are these weights. These weights get learned similar to when we do linear regression, right? We learn all those betas. Same thing here.

[1:48:27] We learn all of these weights. Now, do you guys see how if we have a really big network, there's going to be a lot of weights. So imagine we had a really big network with lots of neurons and they're sending data like there could take take

[1:48:44] sending data like there could take take an image which is an image could be like an image which is an image could be like a th00and by a,000 resolution. be our input and then essentially what we're going to do is take those pass

[1:49:01] them through this network and generate some type of classification of like cat or dog some probability of like might be cat it might be dog might be giraffe. Um so maybe the output's actually three. So we have three final outputs like cat

[1:49:16] probability, dog probability or giraffe probability. But there can be a lot of weights because we're going to end up with this really massive network where everything is connected together, right? So there can be a lot of neurons um a

[1:49:32] lot of weights that are that are connecting layers to each other. So these networks can grow massively with the amount of weights that have to be learned. Um now in this simple model of a neuron there's only three weights only

[1:49:47] three weights that we would have to learn. So pretty simple but in like a learn. So pretty simple but in like a chat GBT there's 100 billion weights that would be pretty conservative actually. Um their opensource model

[1:50:02] their transformer network open source GPT is around 20 billion weights. Okay, so just to give you an idea, there's 20 billion weights that would have to be

[1:50:15] billion weights that would have to be learned. So you can see why substantial resources would be taken to train a model like that. There's going to be 20 billion of these things that have to be learned. Okay, so let me pause here. Any

[1:50:28] questions about this model of a neuron? So, so mathematically model of a neuron, it's just taking a weighted sum of previous layer inputs, passing that through an activation to generate an output from a neuron. That's all it's

[1:50:42] output from a neuron. That's all it's doing. Okay, that and that's that idea is is called the perceptron. That's basically the perceptron is taking a activation. That's essentially the perceptron. We're going to study that

[1:50:57] perceptron. We're going to study that next. But this is the most basic form of neural net computation you can possibly have. Just a single neuron that's taking a weighted sum of inputs and passing it through an activation. So these inputs

[1:51:11] could be features. They could be pixels in an image, right? Many different things. Um but you're just taking the weighted sum of those passing through activation. um and generating an output. One of the

[1:51:27] questions that we should have at this point is what is the activation? It it the activation is actually something that we can change. We can set what we want it to be depending on the problem. So some activations are going to be

[1:51:42] going to be pretty popular in some cases. In other cases, we'll use other activations. We'll actually talk about it later in the lesson. um what this it later in the lesson. um what this function usually looks like um in terms

[1:51:56] of taking that weighted sum passing it through some type of function that function might change from neuron to neuron in the network but right now we're just keeping it generic as some function some nonlinear function that

[1:52:09] this weighted sum goes through. So in that case you know it's different than a regression. A regression would just take this and be done with it right it would just take this weighted sum and that's it. That's the that's the answer. But

[1:52:23] for a neuron, there's one extra component to that, which is taking that and passing it through a function, right? So, so a little bit more involved right? So, so a little bit more involved there. Okay. So, uh everything I just

[1:52:37] there. Okay. So, uh everything I just said um is kind of depicted here where said um is kind of depicted here where we are taking the um the the weighted sum. Now, there may also be a bias. I forgot to mention that. um similar to

[1:52:49] what we had in linear regression, right? There could also be a bias. So what we're really doing here is taking some function. So this this may represent the function that we're doing, some step function um and generating an output

[1:53:04] which kind of mimics the biological idea of an activation of the neuron. Right? of an activation of the neuron. Right? So if the signal is big enough, if it reaches some mathematical threshold, um then the output signal is generated,

[1:53:19] there's some some output [clears throat] that's generated. It's going to be the same here. We're going to be taking a weighted sum

[1:53:32] activation F. So basically what we could do is think of this as some intermediate value Z is the weighted weighted sum and then we're taking the activation of that then we're taking the activation of that Z and generating uh an output value Y

[1:53:46] right which which the activation is going to be something like this uh maybe a nonlinear step function maybe an exponential situation we're going to use different activations. So that is the artificial

[1:54:01] activations. So that is the artificial neuron model um weighted sum through an activation. That's actually it for the neuron. So some terminology in terms of the difference between the artificial model weighted sum through an activation

[1:54:16] um and the actual biological model or the biological neuron. Um the nucleus is known as kind of the node which is the node is just this um the node really is

[1:54:28] just like a a graphical kind of node here that uh we could imagine as part of a network. So a network's going to have lots of nodes uh to it. Um we have our

[1:54:40] inputs which are coming from usually a previous layer in the network. Um but you know in this in the case of a simple neuron like this think of it like our features right our features like our our bedrooms bathrooms

[1:54:55] square footage from our housing data things like that um we have weights which are kind of like our synapses and the biological sense and then we have axon which is the output so there are kind of mirrors to the biological

[1:55:10] definitions. Okay. All right. So let's expand on this idea from a single neuron expand on this idea from a single neuron into a network of neurons. So a neural network then is just the extension of this idea where now we have

[1:55:28] interconnected layers of neurons which is basically our layers of neurons which is basically our computation model for the overall brain. Basically a whole network of biological neurons working together. Um each

[1:55:43] network consists of uh basically an input layer and an output layer and then a bunch of potential layers in between. So you can have as many hidden layers the these layers between the input and output are known as hidden layers and

[1:56:00] there might be many of them potentially many of them. It depends on how deep the network is. A really deep network depth is really based on how many hidden of them, it's going to be a shallow network. If you have a lot of them,

[1:56:15] that's going to be a deep network. So this each one of these blue circles is a this each one of these blue circles is a neuron. So this is a neuron. And you have many neurons connected together, but every neuron's doing the same thing.

[1:56:29] It's taking this weighted sum of of of inputs and passing it through some activation and generating an output, right? That um goes into the next layer. Does the activation of a neuron depend need the inputs to be above a certain

[1:56:44] threshold? Um, biologically yes. But in the in the neural network model, no. Um, the in the neural network model, no. Um, we just activation really just means taking the function and applying it to the weighted sum.

[1:57:04] not required to be above a certain threshold um to to be activated. We we don't really care um from a biological perspective like activated or not. We care more about what is the output which is this quantity.

[1:57:20] We care about this output right uh the the weighted sum uh through the activation. It's just it's a function. Yeah. The activation is really just a nonlinear function. We care about the output. Exactly. We care about the

[1:57:33] output that's coming out of that neuron because what comes out of does it make sense that what comes out of that neuron is really important for the future layers, right? Cuz like what comes out of that neuron is going to influence in

[1:57:46] the network, right? What we produce because what we produce here flows into the next layer and on and on and on and on, right? So whatever we produce from one layer becomes the input to the next layer. in the network. So it really

[1:58:03] matters what comes out of a neuron because that's going to be an input to the next layer. Really really matters what what the output is of every neuron. Yeah. Okay. So just to go back to this definition again, the network is this

[1:58:17] whole thing. So, it's all of these neurons connected together and a a collection of neurons um that are kind of forming an input

[1:58:29] um that are kind of forming an input prior to another layer. Um it this is basically one layer, right? This is a layer of neurons and there and when we build our networks, we can actually control how big that layer is. So, how

[1:58:44] actually specify that when we architect our network. Um that's something we'll be able to do. And how many outputs do we want? So if we're the outputs are we want? So if we're the outputs are usually going to be relative to how many

[1:58:59] classes are we predicting or how many outputs do we generally want. So in like let's say we're predicting between is this image a cat or a dog. Um we we

[1:59:11] would only have these two outputs. So this would be probability of dog and this would be probability of cat let's say or if it was like uh if it was um if we had a giraffe then that would be a third member of the output layer so I

[1:59:27] would need to generate a third for giraffe right or in the case of imageet they needed like in the imageet remember they're predicting a thousand different categories of items so they need an

[1:59:42] output layer that has a thousand neurons in it. So they need a thousand uh neurons in the output because they're predicting a thousand potential predicting a thousand potential categories of images, right? So that

[1:59:57] output layer should match what your final prediction is going to be. So if you're predicting 10 categories, you need an output layer that has 10 neurons which are going to represent the probabilities of each class. If you're

[2:00:09] predicting a regression, you only need one output like the price or the temperature or whatever it is, right? You only need that one output. Or if it's binary like fraud, not fraud, cancer, not cancer, you may only have

[2:00:22] one output, which is the probability of uh of of the like cancer or fraud or whatever it is. Okay. So, how does a neural network process data and make a

[2:00:35] prediction? So, it's kind of what I've been saying. Inputs go through the very been saying. Inputs go through the very beginning layer and then um every neuron that's in that layer receives inputs, assigns a weight, passes that through an

[2:00:48] activation to generate an output. So, every neuron generates an output and the outputs from that first layer are then forwarded to the second layer for processing in the same exact way. So you take a weighted sum of all those inputs

[2:01:03] from that layer, generate an output, pass that along to the next layer and on and on and on. So this network, you know, could have many many many layers. So there could be another layer here that's another what we would call hidden

[2:01:19] layer, which is anything between the input and output. And what we should realize is there's going to be contributions from every neuron that flow into each one of these neurons, right? Right? So there's a lot of

[2:01:31] connections there. Every connection is weighted, right? There's a weight. Every every connection contributes some weight to that neuron. And we process as usual,

[2:01:43] we process a weighted sum through an activation of all of these guys. It's just that their inputs are going to come from this previous layer. Hopefully that makes sense. The inputs are going to come from the outputs of those neurons

[2:01:56] from the previous layer. And on and on and on. So in this way data we would say is kind of flowing forward or what we would call forward propagation.

[2:02:11] We forward propagate because it's it's literally like a propagation like a literally like a propagation like a signal propagation.

[2:02:24] because data is flowing from input all the way to the output going through all these neurons. So that that's how we make a prediction is we forward propagate inputs through the network and generate an output. Right? So that's

[2:02:38] known as forward propagation is sending data through generating a prediction. U which could be a prediction could be probabilities. Right? There's 80% probability this image is a cat. 20% probability this image is a dog. So

[2:02:53] probability this image is a dog. So therefore our final prediction is um cat for instance. So this could be 08 this could be 02 and this would be 08 probability it's a cat and 2 probability it's a dog right something like that.

[2:03:09] And we we're generating that by taking our image and flowing it through forward propagating it through this network generating these probabilities. That's just an example. Okay. Okay. So where we're going to start uh with in terms of

[2:03:24] these networks, the very first one that we've actually already been studying is what is exactly what we've been studying. It's just the model of a neuron of a single neuron. um we will graduate

[2:03:40] of a single neuron. um we will graduate from that into multi-layers of neurons. from that into multi-layers of neurons. So um multi-layer perceptron um so more than one layer, more than one neuron for sure, but potentially more than one

[2:03:52] layer. And then we'll graduate from that into deeper networks that have many layers to them. Uh and then and then once we do that, we'll be in a good position, especially once we learn how to train these guys. um we'll spend a

[2:04:08] lot of effort on training these and optim optimizing that training process and building it within our code like PyTorch or TensorFlow. We will then start to apply variations on these architectures that involve convolutions.

[2:04:25] So the neurons will go from simple weighted sum through an activation to a completely different type of computation that the neurons are doing which would that the neurons are doing which would be like a convolution. So in in the um

[2:04:39] just to go back a step to put this in perspective in this picture we're doing what we would call perceptron style calculations right now meaning we're just taking a weighted sum passing it through an activation and sending that

[2:04:53] forward through the network. When we get into computer vision and working with into computer vision and working with text and RNN's these neurons will now be way more advanced. the neurons will do things like convolve or do convolutions

[2:05:09] which will be um a very sophisticated computational operation that will computational operation that will require a GPU or else will take forever to train any models. So we will graduate from these very basic neurons. So right

[2:05:23] now we're just talking about a basic neuron that's doing this weighted sum through an activation and we'll graduate into convolutions and then we'll into convolutions and then we'll graduate into RNN. So doing recurrent

[2:05:37] calculations uh which mimic kind of sequence memory um for for learning how to work with text sequences like sentences. So, a lot to cover obviously

[2:05:49] before we get there, but right now we're starting with the most basic model of a neuron, which is this perceptron kind of style calculation. Uh, just sending a weighted sum through an activation. That's it. But eventually,

[2:06:03] these neurons will become more and more complex like doing convolutions for complex like doing convolutions for image data or doing RNNs for uh text image data or doing RNNs for uh text data and eventually transformers. Um so

[2:06:16] so these neurons will do even more advanced things like attention in a advanced things like attention in a transformer.

[2:06:29] about the uh you know the perceptron um being the simplest type of network. That's where we're going to start. It's mainly used for binary prediction. So because like we can just produce that one output which could represent a

[2:06:45] probability. So the perceptron was the simple model of the neuron and it was originally used for binary classification problems. So that's where we'll start is the same thing kind of for binary. Um and the perceptron's

[2:07:00] really only good if your data is linearly separable. We'll talk about what that means and why that's really only the case. Otherwise, we need to go to more advanced like more layers. So, it the perceptron is going to be kind of

[2:07:16] limited in the sense that it's a single neuron. It's not a network. So, it's not going to model too many problems that well. Um, but it is the foundation for building on that and generating many many layers of neurons that that we will

[2:07:32] see for larger networks. So, and that's where we go with the MLP, which is multi-layer perceptron. Um, basically multiple layers of these uh perceptrons

[2:07:44] of these neuron models and uh it can handle a little bit more complexity by learning nonlinear things. Um, because it has multiple layers passing those inputs through these nonlinear activations, it can learn more

[2:07:59] activations, it can learn more effective. However, the MLP is not necessarily a deep network. So, it's not going to have that many layers of perceptrons, maybe only a handful. It's not going to be that deep of a network.

[2:08:13] Um, so itself will be kind of limited and it especially won't work well for like images or text, but for maybe certain problems like our housing data might work okay. But we'll certainly graduate from perceptron to multi-layer

[2:08:30] perceptron as a little bit bigger of a network. Okay. And then we'll work our way up to deep neural networks which are going to have many hidden layers. So this we we've had this image before. Here it is again where you know a deep

[2:08:45] Here it is again where you know a deep network's going to have um this is the depth here. How many hidden layers do you have? A deep network is going to you have? A deep network is going to have many many many layers. Okay, so

[2:08:59] again like modern generative AI neural networks are going to have thousands of layers, thousands of these hidden layers here. And and notice how they're all connected because you know the inputs the sorry the outputs of one layer

[2:09:15] become the inputs to the next layer. So you can see connections from each of these um contributing to that neuron. This neuron is taking a weighted sum of those and then generating an output

[2:09:29] and then there's connections from that output to every other uh neuron and on and on and on. Right? So this big network you get um a certain number of outputs a certain number of outputs that um mimic what you're trying to predict.

[2:09:44] So again there's like this could be a thousand for the image net problem. Um let's say we're doing cat, dog or giraffe. This would only be three sending an image through forward propagating it generating these

[2:09:58] probabilities which is our prediction. That's kind of how it works. Is it always left to right or can it bounce back? So it is left to right for the networks that we're talking about. The only um time you get the circle like is

[2:10:11] only um time you get the circle like is a recurrent neural net and RNN that is the so the circle you're describing is the essence of a recurrent connection. So it has the ability to do to do this which is kind of like mimicking a memory

[2:10:27] of a sequence sort of like going back and and remembering what our previous uh inputs were. Um otherwise it's going to flow forward. It's going to be forward propagation left to right. But in an RNN, you would get connections like this

[2:10:42] that's what you were talking about, those recurrent connections. We would get those in an RNN. Yeah. So, not Yeah, we won't have those yet. Not until we get to RNN's later in the in the lessons. So, and then the the

[2:10:58] the So, we'll we're going to study this. We're going to work our way up to this. And then as I said once we learn how to build these guys these deep neural networks and really work with them in PyTorch and TensorFlow we'll graduate to

[2:11:13] CNN's the convolutional neural nets which was um this was um motivated by mimicking kind of like the visual reception of the of the eyes and the reception of the of the eyes and the cortex. Um using the convolutions to uh

[2:11:29] to accomplish that. Convolution being a um a type of computation will basically replace those neurons with more advanced computation. computation. And the um visual reception is kind of

[2:11:43] mimicked by the convolution allowing us inside of images to pick out things like objects, things like shapes, edges, lines, um human uh features like faces, allowing us to learn those features of

[2:12:00] an image. So the convolutional net again this was that Alex net 2012 really revolutionized being able to learn uh how to predict on image data. So we will

[2:12:13] learn about CNN's after we cover uh deep neural nets. Uh it stands for I'm not sure what it means in this case actually. Not 100% sure.

[2:12:25] Yeah not not 100% sure. I'd have to uh look that up. I think they're just look that up. I think they're just different um sections of the brain. I think they're they're short for like different uh sections that process like

[2:12:38] uh visuals. Okay, there we go. Yeah, they were different regions. Inferior temporal cortex. Nice. Okay, that makes sense to me. Okay. Okay, so then talking

[2:12:50] sense to me. Okay. Okay, so then talking about RNN's really quick. Um, so RNN's are a lot, you know, networks that we're going to study later on in one of our lessons after we cover deep networks that will handle sequential data, right?

[2:13:04] So this is going to be things like language because as I said, that's a sequence of words after the other in a in a sequence. Um so we will handle text in a sequence. Um so we will handle text with RNN's also um a a very popular use

[2:13:19] case for RNN's especially these days is like forecasting so time series data like uh sales or revenue day by day by day you can you can build forecast with neural uh RNN's um because they're really effective at sequence learning

[2:13:35] really effective at sequence learning mainly because they um allow for these recurrent connections uh So they kind of have this short-term memory which is effective for some language tasks and for forecasting.

[2:13:48] Unfortunately that short-term memory is is too short. It's not able to learn the type of context that we see in like a a chat GBT for instance with a very long

[2:14:00] prompt or a lot of text which is what we see in transformers. So RNN's is kind of limited relative to transformers on language tasks mainly because that what it needs to [clears throat] be to generate text or do effective

[2:14:15] translations things like that. So we're going to learn about that when we cover RNN's. Um one of the things that I mentioned that's certainly true is when mentioned that's certainly true is when we uh work with neural nets we will be

[2:14:28] able to customize basically everything about that network. how many layers there are, how many neurons are within each layer, what those activation functions are. So you can think of like you know this is going to be completely

[2:14:45] customizable from the same with input and output. How many uh inputs are are there? How many outputs should there be? Like if I'm predicting three or 10

[2:14:57] classes or maybe two or maybe just one in a binary case. Um, basically everything about this is going to be customizable when we build our models. And and that's just what we would call the architecture of the network, right?

[2:15:12] How many layers there are, how many neurons there are within each layer. And you can see that like when you have a deeper network with a lot of different

[2:15:24] uh layers here. So like this example on the right, you can see this is going to have a lot more um this has a lot more weights to it that have to be learned. So it's going to take more data. It's going to take longer to train a model

[2:15:40] like this cuz there's so many more connections, right? There's so many more weights um that have to be learned during this process um than than if you during this process um than than if you had fewer layers. So, um, that's a

[2:15:55] trade-off, right? Maybe you need more and more layers to learn a really complex problem that may be necessary, it comes to computation because there's just going to be more weights that you

[2:16:09] have to keep track of, right? And and and learn. Um, so this is something we'll see when we get into TensorFlow and PyTorch. We're going to see how do we build this? What's the code that generates this kind of network? That's

[2:16:24] What's the code that does this? That's something we're going to learn. It's it's actually going to be relatively easy inside of TensorFlow and PyTorch. They make it easy because those libraries are so good at building it.

[2:16:38] So, not only that, like we'll control the architecture, but we'll also control the training and the evaluation. So all of that we'll be able to set up um inside of our our you know PyTorch or TensorFlow when we get to that. Okay. So

[2:16:53] one of the things that's true about neural nets is they typically require neural nets is they typically require more data and um there's uh that's kind of a graphic here to show that like if you have even a small neural network

[2:17:07] that's very shallow. So small being like number of hidden layers you know it can still outperform traditional machine learning. So think of like logistic regression, decision tree, random forest, xg boost. It can still

[2:17:21] outperform all of those. It's just going to require more data. And the larger the model, the more data that you're going to need. So you can kind of see that on this graph, right? The large neural network has a lot of potential to

[2:17:35] network has a lot of potential to perform really, really well. Um, but it needs the most data. it's the larger that the network is, the more data it needs to to function properly. Now, these are relatively close. So, just a

[2:17:48] these are relatively close. So, just a shallow network isn't and especially in the small data um down here, it it's not the small data um down here, it it's not until this point that um the the even

[2:18:00] the shallow neural network kind of surpasses traditional machine learning. surpasses traditional machine learning. So again for small data um it may be desirable to just continue to use traditional techniques like what we've

[2:18:12] studied but as we get larger and larger data size um if we're working especially with unstructured like images text we're going to need neural nets and um you

[2:18:24] know usually we'll start small and work our way up to larger neural networks if we need them you know in terms of the architecture which is how How many layers do we have? How many neurons do we have? Those kind of things. Okay. So

[2:18:38] we have? Those kind of things. Okay. So the the depth of the network like how the the depth of the network like how many layers we have is often um relative to the task. So if we're doing, you know, working with images like in an

[2:18:51] image classification or like object detection or like in a in a transformer for a generative AI use case, that's going to be a much bigger network than maybe a simpler modeling task like predicting the price on our housing data

[2:19:06] where it's very structured, has a certain number of features to it, we may not need as deep of a network. So this is again where we will uh we will you

[2:19:19] know practice building networks and seeing when that makes a difference like how much we change that architecture uh on our modeling problems. We're going to see that as we get into it. But here's a

[2:19:33] good you know this is a good picture here on the uh this one has two layers here on the uh this one has two layers and it produces two outputs. Um so it takes four input features produces um two outputs which could be probabilities

[2:19:48] let's say and you can see like these these symbols are exactly what I've been describing as you take a weighted sum and you pass it through an activation. That's what the sigma and the f is. So the sigma is like the weighted sum of

[2:20:02] the inputs to that neuron. Every one of these circles is a neuron, right? So this is a neuron. This is a neuron. This is a neuron. neuron neuron neuron neuron and um technically all these

[2:20:19] other circles are neurons as well. These are just output neurons and these are input neurons. So they're not really doing much. They're just processing the doing much. They're just processing the input or holding the output. But but

[2:20:31] they're going to be a weighted sum through an activation for all these neurons, right? Weighted sum through an activation. That's exactly what what this is meant to represent. Sigma weighted sum through the f through the

[2:20:44] activation. Um so let's talk about the perceptron uh as a a model that is based on that neuron that we've studied so far. Um basically one and the same. So

[2:20:59] what is what do we mean by perceptron exactly? So I've been using the word perceptron a little bit. Let's actually define what it is. Essentially it is a very basic neural network that is designed for binary classification. So

[2:21:13] think spam not spam right fraud not fraud. So it has only two choices. And fraud. So it has only two choices. And essentially what it is is the neuron such that it has a very simple activation. This is the activation right

[2:21:27] activation. This is the activation right here which is that um you get a a here which is that um you get a a classification of one if your weighted sum times your your inputs plus your bias is positive. So this has an

[2:21:40] activation function like this which is basically just checking is your weighted sum uh is your weighted sum times your inputs or weighted sum with your inputs

[2:21:53] um greater than zero. So if it's greater than zero, we're going to output for than zero, we're going to output for that neuron a one. If it's negative or equal to zero, we're going to output zero. So it's in that way, it's just

[2:22:06] taking that model of the neuron. So, you're taking all of your inputs, your X, X1, X2, X3, X, XN, however many you have,

[2:22:18] and you're feeding all of them into the single neuron, single neuron, which is taking a weighted sum and uh taking a weighted sum of those inputs and then passing it through an

[2:22:31] activation, we're just going to produce a a zero or a one, right? um a y which a a zero or a one, right? um a y which is zero or one depending on this is zero or one depending on this activation which is um was it positive

[2:22:46] or was it zero or negative right is essentially what this activation is. So it's a very simple model used for binary classification that's basically just a classification that's basically just a neuron um just a single neuron taking

[2:23:00] that weighted sum uh passing it through the activation. This is the activation activation function here which is basically just checking is it positive

[2:23:12] basically just checking is it positive you get a one output uh is it negative or zero you get a zero output right for the for the neuron that's all it is. So perceptron a very simple model um that is based on this neuron calculation. The

[2:23:26] thing that's unique about the perceptron is it's just using this particular activation um to see you know do we get a one or a zero for a binary classification. So what's interesting about the perceptron

[2:23:39] what's interesting about the perceptron is um you know it's uh using the basic taking a weighted sum through an activation. And what gets learned in

[2:23:51] we've talked about learning in any neural network, which are these weights, right? These weights are going to be what gets updated through the training of the perceptron. So, we need to talk about how does that happen?

[2:24:06] What kind of adjustments do you make? Um but essentially we're the algorithm is going to learn those weights of that neuron um so that it can produce good uh

[2:24:19] outputs for your data, right? You want highquality predictions that line up to your your like spam not spam fraud not fraud kind of uh uh predictions. So it's

[2:24:31] exactly like this picture. uh it's just the output again is going to be a binary zero or a one and that activation is going to be particularly checking if it's positive or if it's negative or zero. So it's exactly this picture we

[2:24:46] just talked about from earlier with the neuron. Um for that reason you know the weighted sum through an activation is often just referred to as a perceptron style computation because that's essentially what it is. The perceptron

[2:25:02] is just using that particular activation of seeing if that weighted sum is positive. So it's that very particular activation. Again, I'll draw it here. activation. Again, I'll draw it here. you know we we do this uh weighted sum

[2:25:18] and we're passing it through the the activation and the activation is such activation and the activation is such that um if this weighted sum that um if this weighted sum is positive then you get um then you get

[2:25:31] is positive then you get um then you get a a one um if it's uh else basically you a a one um if it's uh else basically you get a zero as the output right is that's the activation function there. So the things that need to be updated and

[2:25:46] learned to have an effective perceptron is obviously these weights. These play a really critical role uh in determining what the output is going to be in any prediction. So again what does this look like? Um it's very much similar to

[2:26:02] linear regression. We talked about this earlier and for that reason this model only works really effectively when the data is basically what we would call

[2:26:14] linearly separable. So if our classes like let's say we just had two features X1 and X2 um if we had let's say these red points belong to one class and then these blue points um or let me

[2:26:31] draw them in green belong to a different class over here. If you know if we could draw a straight line through here and say everything on this side belongs to yes and everything on this side belongs to no those binary classes then this

[2:26:47] model would be effective because it's essentially going to find that line and compare it to zero essentially as if we shifted this line to be kind of at the origin and we're checking like does this result end up positive meaning it's on

[2:27:03] this side of the line or does it end up negative, meaning it's kind of on this side of the line, which would be a zero. Um, so the perceptron is really only good to be used because it's so simple, just a single neuron. It's really only

[2:27:19] good in these situations where the data is kind of what we would call linearly separable, which is not always the case, right? That's typically not the case. Um which is why this model we need to expand this to have many neurons with

[2:27:34] expand this to have many neurons with nonlinear activations to model like real world problems. Right? So again the perceptron uh we get a zero or one. We're going to adjust the weights during the training process. um they get

[2:27:49] updated kind of in a gradient descent style uh calculation that happens to adjust those according to the error and the perceptron really only works well uh when the when the data is linear separable as we just talked about. Okay,

[2:28:04] so it's everything here that we just talked about weighted sum through an activation. It's just this prediction is now we know what it is for a perceptron. We know that this is going to be a zero or a one.

[2:28:17] going to be. It's it's the you know checking if that weighted sum is checking if that weighted sum is positive or negative or zero. Right? So we know what these components are now for the perceptron. So everything we've

[2:28:30] talked about is is here. Nothing really changes except a particular activation for this binary case and it's just a single neuron. So that's the other thing we've been talking about. A network of neurons. The perceptron is so basic.

[2:28:45] just a single neuron. All right, just a single one. It doesn't have a layer of multiple neurons. Um it's just a single one. So very basic. So the components there are again what we've been talking

[2:28:59] about. They we have a set of inputs. We have a set of weights. We have a bias using for shifting um uh which is important for checking relative to zero.

[2:29:11] Um we have a summation function that uh just takes the weighted uh sum and then we have an activation which is exactly what's producing our zero or one right the activation everything is there that we typically will have in any neuron and

[2:29:28] certainly in a network of neurons we're going to have lots of these guys. Okay, going to have lots of these guys. Okay, so the perceptron is a good model of a single neuron um in the sense that it takes a weighted sum, takes an

[2:29:43] activation and produces an output. Now what would happen with this that output what would happen with this that output in a in a larger network is we know that data would flow forward to the next layer, right? We know that um in a

[2:29:56] larger network, but in the perceptron, we really only just have this, right? we just have some weighted inputs going into that single neuron. Um, but typically that data would flow forward in what we've said is forward

[2:30:11] propagation. Another word for this for uh type of network where data only flows forward in a forward propagation manner is um sometimes known as a feed forward

[2:30:24] feed forward network. So, so feed forward is basically just a network of perceptron neurons, right? That that perceptron being um their computation is

[2:30:36] activation and it's just flowing forward. The output goes into the next layer. That output goes into the next layer and on and on. So this is sometimes known as just a feed forward network because we do later on when we

[2:30:51] have RNN's we'll have the ability to have a recurrent connection that goes backwards but right now you know these especially with perceptron style calculations in these neurons um everything here is just what we call

[2:31:07] feed forward um everything just forward propagates nothing nothing has a kind of feedback back connection yet until we get to RNN's. So, this is another term

[2:31:19] you may hear is a is what we call a feed forward network. It's really just um whenever you hear the word feed forward network, you should think of perceptron style computations that just flow forward weighted sum through an

[2:31:33] activation going forward. That's all passing their output as the input to the next layer. So, some of the features of this is that information only goes one way. um it goes straight it goes forward propagated through never touches a node

[2:31:47] propagated through never touches a node twice. It has no memory of the input um We would have memory because we care about sequences. We care about data that's prior to our current um value that we're processing in the sequence.

[2:32:03] Um so h so the the feed forward network which is the most basic type that perceptron style computation data just flowing forward basically has no

[2:32:15] flowing forward basically has no ordering no sequential memory nothing like that because it cannot go backwards it's just every all the data is um flowing forward okay so what we're going to do now is just look at the extension

[2:32:29] of the perceptron so given that we have a perceptron remember The perceptron is a perceptron remember The perceptron is just a single neuron that has inputs. So we have inputs flowing into a single neuron which is generating a single

[2:32:41] output. That's perceptron is just this. It's not even a network really. It's just inputs uh weighted some of those going forward into an output. However, going forward into an output. However, if we stack together multiple layers of

[2:32:56] if we stack together multiple layers of these guys in terms of these neurons, we can generate what's called a multi-layer perceptron, which is starting to be a very basic network, right? It's not a full deep network in the sense there's

[2:33:12] not many many layers. There's not many many neurons. It's very limited. So multi-layer perceptron is designed to be pretty limited in terms of its structure. It's not a huge network. It's mainly just a small collection of maybe

[2:33:31] two, three, maybe four at most perceptron layers. Um and and we generate an output which could be one or two or three outputs. Depends on what we're trying to predict. But it's think of just stacking these neurons together

[2:33:47] in a simple collection of layers and we're starting to build a network that we're starting to build a network that is the multi-layer perceptron. Um now every neuron in the MLP sometimes this is known as MLP multilayer perceptron.

[2:34:02] Um every layer in this multi-layer perceptron is just the simple weighted sum through an activation. So this is still every single neuron still functions the same weighted sum through the activation, right? It's it truly is

[2:34:17] that weighted sum through an activation. So nothing really changes there. Every neuron is like that. Okay? And all the data still feeds forward. It's just now we have multiple neurons and multiple layers, maybe two or three or four of

[2:34:33] layers, maybe two or three or four of them. Okay? So a perceptron them. Okay? So a perceptron um is decent except it fails at the linearly separable. So a very basic example people like to give of where the

[2:34:47] neuron model of a perceptron really falls apart is like a simple um what's falls apart is like a simple um what's called an xor which is uh the like the called an xor which is uh the like the exclusive or operation between uh um

[2:35:01] like two binary values like minus1 and one minus1 and minus1. Um if you take the exclusive or between the two you can see what their values are. Like the exclusive or between minus1 and minus1 is still minus1. Um the exclusive or

[2:35:16] between minus1 and one is one. And then um what's interesting is you know this is not uh separable. You would have to in order to separate these two classes, you would actually need two lines or

[2:35:31] basically something nonlinear. Uh because you would need um you would because you would need um you would basically need to draw something uh kind of like this um

[2:35:44] in order to separate the two uh and say like everything on this side is uh belongs to one class and everything on this side belongs to another class. And that's highly nonlinear. So, and this is just one example of a function that

[2:35:58] would break the the you can think of many more like you could have like let's say um going back to the example we could draw circles here and here and

[2:36:10] then draw um the [clears throat] red ones um like this one could be on this side and this one could be on this side. And again, this makes it really you can't draw a single line through there and say, "Well, I can perfectly

[2:36:25] separate everything on this side and everything on this side." That's not going to be possible with with a function like this with data that's like function like this with data that's like this. Um, so that's just to say the

[2:36:37] perceptron is not perfect. It's going to struggle on data that is not linearly separable, which is what this means. meaning you can draw a perfect line or a hyper plane and all the data is on one side of it and all the other classes are

[2:36:52] on the other side of it for a binary classification. Um you you most likely classification. Um you you most likely need a multi-layer perceptron because you need nonlinearity, right? You need this this kind of

[2:37:05] picture here in order to better separate it which is which is nonlinear. Okay, so that's just to say a single neuron by itself is kind of limited. That's where you need layers of neurons to more effectively model problems. That's all

[2:37:21] demo for you guys and this is just going to use um so it turns out that perceptron a single neuron is actually available within scikitlearn. So we don't even need to use tensorflow yet or pietorch

[2:37:37] yet in order to just work with perceptron. Um, so there's actually, just to show you that quickly, there is a a demo. So this 3.04 demo inside of um, lesson three here, which let me go over to that

[2:37:55] and show you. So, we'll do this demo. There should be a couple in lesson 3. We're going to do this 3.04. I'll give you a moment to pull it up. It should be in your uh lesson three materials. And there's also

[2:38:11] a I believe a data set that you're going to need which is the spam the spam based data set which comes in the lesson 3 data sets. I can upload it here. But do you guys have the do you have this notebook from the lesson 3? You might

[2:38:26] need to download it from your reference materials. Okay, great. Okay, so let's um let's practice building just the perceptron. Now again, would we typically use a perceptron? No, we would typically use a a uh neural net that has

[2:38:43] more than just a perceptron, which is a single neuron. So, but the perceptron is single neuron. So, but the perceptron is an important building block to larger and deeper networks because it that fundamental idea of taking a weighted

[2:38:58] sum through an activation is really critical and it's it shows up in a lot of networks. So it it is an important building block, but by itself is not especially in those cases we just saw where it's not linearly separable.

[2:39:14] Let's see how we use the perceptron here. Um what you can see is look at where it comes from. It's from scikitlearn in the linear model family. Now we've seen other things from there too like the uh logistic regression,

[2:39:32] there. But for classification, we've seen logistic regression from that linear model. Here is perceptron. It's considered to be in this linear model family because we're taking that weighted sum which is linear but then

[2:39:47] passing it through an activation which is uh checking if it's positive or or negative. Right? So let's load let's run that to do our imports. Okay,

[2:39:59] let's load our data. So, we're going to load that spam based data. I do need to load that spam based data. I do need to put it in here.

[2:40:12] Okay. So, that's loaded and we can check the uh head of it to see the first five rows, of course. So, you can see there's rows, of course. So, you can see there's some uh some data. Now, this the spam is

[2:40:25] is whether or not the email was spam or not. So it's got a it's all a bunch of not. So it's got a it's all a bunch of data about the uh about the email and then it has a classification category of spam which is the last column which is a

[2:40:40] one or a zero for not spam right so that's that this is a classification problem uh and that and this data is just a bunch of features about different words so for instance this is a word frequency of the word all the word

[2:40:55] address the word make the word power, over, remove. So, it's got a bunch of word frequency features, and then it's got some capital capitalization features, um, characters like the frequency of the

[2:41:10] pound symbol, frequency of the dollar symbol, frequency of exclamation point, symbol, frequency of exclamation point, um, parenthesis. So, it's got a bunch of um, parenthesis. So, it's got a bunch of features about text uh, within the, uh,

[2:41:23] email. Okay. All right. So, what we're going to do is um just have a simple check here. If there are any nles, we're just going to fill those with zeros. So, um obviously,

[2:41:39] we could do a more robust check here. Um in fact, let's let's check uh you know, in fact, let's let's check uh you know, we know how to do dataisnull and then dot sum. So we can see that across all the columns. Uh if we have

[2:41:54] any um doesn't look like we have any. So this yeah it looks like we don't have any nles. So this should be relatively uh this shouldn't do anything essentially cuz this should fail. There

[2:42:09] are none. So this should be false. It shouldn't have to fill in anything. So that that's fine. That's just going to produce our data.

[2:42:22] the important part. This is the important part of grabbing our features and our labels, right? So this is grabbing our features and our labels, right? Features and

[2:42:37] labels. So this is grabbing every row and every column but the last one. That's what I look does, right? Do you guys remember that for our data frame? This is grabbing every single row but then every column but the last. So

[2:42:53] everything up until the last which is the minus one index. That is all of our the minus one index. That is all of our features. And this is grabbing the very last column all all rows. That's what the colon means. All rows. And then only

[2:43:07] the last column which is that's the label right? This is the lab the spam label right? This is the lab the spam label uh column is the last and we saw

[2:43:19] that in the in in this uh head of the data frame right spam is the last one. Every other column before it is is a feature. So we are just separating those out. Pretty typical

[2:43:39] doing what we usually do with that is train test split, right? Pretty standard where we pass in our features and our labels and we do a train test split. Um, and we can choose whatever test size. I think point4 is a little big. We could

[2:43:53] think point4 is a little big. We could lower that to maybe 3 or 0.25 or 0.2. Um, but I'll just leave it for that for now. Um, but this is our usual train test split. So nothing that interesting here.

[2:44:05] just generating our our training data and our testing data. It's pretty standard, right? Nothing that uh out of the ordinary there for a supervised problem, which this certainly is. Even though we're using perceptron, which is

[2:44:19] a neuron model to do it, still supervised. Let's scale our data. So, we're going to scale all of our features. Um that's uh

[2:44:31] still something we're going to do here. um using this perceptron uh scaling all training features, scaling our test features.

[2:44:48] both of those and then we can fit our perceptron. So look how easy it is to initialize a perceptron. Um and and uh basically the

[2:45:01] perceptron. Um and and uh basically the perceptron is um just a single neuron model, right? So that's all it is. And it's going to use um uh it's going to it's going to use um uh it's going to train. So this uh this ffit, I should

[2:45:15] train. So this uh this ffit, I should call this out. This ffit is going to call this out. This ffit is going to train the neuron which is just the perceptron just a single neuron. It's going to train this by learning the

[2:45:30] weights through a gradient descent uh algorithm. So it's going to run a gradient descent in the background to learn what those optimal weights are. Now we haven't

[2:45:45] studied gradient descent in detail. We will as part of like when we get to the will as part of like when we get to the part on uh coming up shortly on learning how to train neural nets. We're going to learn a lot about this because every

[2:45:58] single neural net trains through gradient descent algorithm one way or the other. It does gradient descent. So it's not going to be any different for the perceptron because it is just a neuron. It's a very

[2:46:11] basic. It's not a network per se. But it is a neuron. The network is just a huge collection of neurons. So, um they're all going to train through gradient descent. So, we'll we'll learn more about that as we go. But this ffit is

[2:46:25] about that as we go. But this ffit is going to run gradient descent on our um our our features and our labels here for the training data. And it's going to the perceptron is going to adjust all of its weights um according to this data.

[2:46:41] Okay. So we get a perceptron that has been fit. Um and then we can make predictions. So we can pass in our training data to get a collection of predictions for our training set. And then more importantly, we want to

[2:46:56] evaluate that on our test set. So we pass in our test features to our model and dopredict. So so far this is just behaving like any other scikitlearn model, right? Nothing that interesting. But it is cool that

[2:47:11] the perceptron exists inside of scikitlearn as kind of like any other model. Okay, so we can see the accuracy here on our data and we get about an 89% accuracy on the training and about an

[2:47:25] 88% on the test. So pretty decent. That's not too bad. Um using that perceptron. So pretty cool. Um that works out. And that's just a that's just a simple neuron. It's not, you know, this is a

[2:47:39] very basic model. Um, it's just that single neuron. Of course, where we're going with all this is to build up our own neural network that's going to be a huge collection of these neurons, right? That are going to be um organized into

[2:47:54] layers that are connected into other layers and so on and so on and so on. So, we're going to build up to that. And one of the things we're going to have to learn is how to construct that in like PyTorch or TensorFlow. Um, so that's

[2:48:08] also going to learn how to set up this training algorithm, this gradient descent. We need to learn more about that. So that'll be upcoming as well. Okay. And we covered this, but I just wanted to start here with kind of a

[2:48:21] review of what we had talked about in terms of uh neural networks, right? So just to refresh ourselves, we're dealing with uh neural networks in general and

[2:48:33] um we know that those are comprised of kind of a network of neurons, right? And and um these are organized into layers where every layer has a different number of neurons potentially that we could we could specify how many there's going to

[2:48:46] be. But the the idea is that the data that we pass in uh will kind of flow forward through this network and every neuron is really doing something like neuron is really doing something like this. Remember this is and this is the

[2:49:01] perceptron which is that uh we take a weighted sum which is that uh we take a weighted sum times the input. Um, and so this W star

[2:49:14] X is really meant to be um kind of a dotproduct like a weighted sum uh times inputs, right? So we have weights times inputs and then we um might add a bias

[2:49:26] in there as well, which is what the B is here. So adding in a bias, which is just another weight really. And but the important thing is passing that weighted sum of inputs through an activation which is this F, right? So we pass that

[2:49:41] through some type of activation which generates the output for that neuron and what happens in the network is these outputs just flow into the next layer. So whatever is output from from this neuron becomes the input into this

[2:49:57] next neuron. It also becomes the input into this neuron and and this one really layer. And of course those have different weights associated to them. So

[2:50:09] there'll be a weight here, a weight here, a weight here that that dictates how much of that input from that neuron gets contributed to to this neuron's uh output. And same with this guy, like this guy's going to contribute a certain

[2:50:23] amount to this. This guy's going to contribute a certain amount to this. And there's going to be some type of weighted sum through an activation. And that goes from every layer to every layer. Right? So um that's what we've

[2:50:36] kind of learned so far. The things that we need to study a little bit and we will coming up next are what are these activations um in particular and when do

[2:50:48] we use certain activations. So in certain situations um you know a certain activation will make more sense because they can if you look at the activation from a math perspective they really control what the output of this is

[2:51:01] allowed to be from a range perspective right because you're taking us an input passing it through this function and generating an output. So depending on what that activation function is kind of controls what you can possibly get out

[2:51:15] of there. If you're predicting things like in a in a probability sense, you that's going to limit your range to a probability like between 0 to one. That makes sense. Um if you're doing a regression, you don't want to do that.

[2:51:29] You usually want it to be just any real number value, right? To to mimic like a price or temperature or something like that you're doing a regression with. So this activation is really important and we're going to talk about some very

[2:51:42] common activation functions that are out there. But this is what we're talking about with neurons, right? Weighted sum through an activation and we're doing this in this network of neurons uh that are kind of spread out amongst these

[2:51:55] layers passing data forward throughout and you finally get to some output. And even the output, remember the output when we get into larger and larger complex use cases, the output is going to be different depending on what kind

[2:52:10] of problem we're solving. So if we're um just predicting like a binary output, output that's a probability, right? Between 0 to one, kind of like what we saw for logistic regression for like a, you know, spam not spam, fraud, not

[2:52:26] fraud kind of use case. But if we're predicting maybe we we're predicting categories like this is an image of a cat, this is a dog, this is a giraffe. Um we would we would likely have three outputs, right? We'd have probability of

[2:52:41] dog, probability of cat, and probability of giraffe. Or in you know, if we had 10, then we'd have 10 outputs here. Um so our output layer should kind of match how many outputs we're trying to produce. Um that makes sense. And so

[2:52:56] use cases. Want to set up those output layers to contain the right number of outputs and also you know actually be probabilities which is going to be hugely dictated by

[2:53:09] this uh activation right we want the activation to generate probabilities for those neurons. Okay, so that's a little bit of a recap. And if you guys remember um perceptron is a a special type of uh neuron. It

[2:53:26] that is just a single neuron that has an activation function that produces a zero or one, right? It's it's like a step function that produces a zero or one. Um depending on if this weighted sum is positive, this is going to produce a

[2:53:42] one. If it's negative, it'll produce uh a zero. So that's what we meant by the perceptron model. It's just a single neuron that does this activation. Hopefully that rings a bell from last week. And and what we're really

[2:53:56] bigger neural networks is basically having many perceptrons, which are basically all of these models of neurons here. So you have lots of perceptron style uh calculations and that's where

[2:54:10] we we graduate from just a single neuron to multiple layers of perceptron which is which is this MLP model multi-layer perceptron model here. So that's that's is going to keep expanding. So we're going to have even more layers, even

[2:54:26] more neurons, and then eventually even more complex computations there that are not just weighted sum through an activation, but things like convolution, things like recurrent layers. So we're going to work our way up there in the

[2:54:39] future, but that's kind of where that's where we're headed with all this. So just wanted to recap that a little bit um and pick it up from there, especially the the activations. I think that'll uh we'll talk about that next. So we we

[2:54:53] ended on this demo last time which was um kind of going into scikitlearn if you remember we did this demo went into scikitlearn and scikitlearn actually has scikitlearn and scikitlearn actually has the basic perceptron um model um but of

[2:55:06] neural network it's going to be in tensorflow or u pietorch but um scikitlearn itself does have um a perceptron um because it's it's a very basic model of a neuron um but it's not really a

[2:55:21] network. It's just kind of a single neuron. Okay, so before we go into activations or I really want to pick it up here um and talk more about uh what

[2:55:33] and what are some good examples of activations. Let's talk about activations then. So uh we know what the definition is uh in in terms of it is a function that we're going to apply to that uh inputs that are generated for

[2:55:50] the neuron where we have that weighted sum uh with weights times inputs and that generates this value that we're going to pass through this function calling that function the activation function which gets the name activation

[2:56:04] function really comes from biology and and kind of the actual um workings of a neuron which uh gets activated if there's enough kind of energy there. So same kind of thing here if if the input is enough we run that through a function

[2:56:20] and we kind of generate this uh output of it through this activation function and so that's where it gets kind of its name. Now in terms of the perceptron we know what the activation function is. It's it's actually um a very simple

[2:56:34] function which looks at looks at this weighted sum of inputs. So it takes this um weighted sum. So you can see here's like an input one, input two all the way up to input in. We wait those. So there's a corresponding weight for every

[2:56:49] one of those inputs. We know that. We total that up and maybe even add in a bias. So you can think of an extra term here. there's a bias that it get uh

[2:57:01] here. there's a bias that it get uh contributes to the sum typically and then this goes through this f right this f ofx um activation which is taking the f ofx um activation which is taking the f of that uh weighted sum uh that is all

[2:57:17] of these guys right so it this activation function is being applied to activation function is being applied to those and really in the perceptron it's a simple function the f really says that If so the the f um is such that um it

[2:57:33] If so the the f um is such that um it has a definition such that like if um has a definition such that like if um this weighted sum is positive uh then we this weighted sum is positive uh then we produce um if this sum is positive then

[2:57:45] produce um if this sum is positive then we produce uh one um else we produce uh zero else being you know it's negative or zero then we produce zero right so the activation for the perceptron is really really simple. But that that's

[2:58:01] there. And in fact, when we start building our own neural networks, um there's going to be many different activations we can choose from. Some of them are going to be more um popular than others, and some of them have very

[2:58:14] particular use cases. As I said, one thing you want to pay attention to is this activation really helping us kind of narrow down the range of what the output of that neuron could possibly be. So like in the perceptron case, the

[2:58:30] output's only allowed to be a binary one or a zero, which is good for certain problems, right? It's good for like a spam not spam cancer not cancer, but all. Right? It wouldn't be useful at all for regression. wouldn't be useful at

[2:58:46] all for generating a probability even because it's still structured to just zero or one. Uh and it certainly wouldn't be good for multiclass where we have maybe 10 different categories of things we're predicting. Um so so the

[2:59:01] perceptron activation is very limited. Um but we're going to explore some other ones that are found in a lot of lot more general neural networks than just perceptron. But but you see what its purpose is really to produce the output

[2:59:16] of a neuron um passing in those inputs through this activation function. So it has that capability of kind of um mapping those to a particular range which is important in terms of what are we trying to get out of the neural net

[2:59:30] especially as you get towards the the output layers right then you really care about what your final output is. Okay, so if you think about it, if like what activation function? So just think about that for a second. If we didn't have

[2:59:46] that um then in theory like you know depending on what those inputs are, we really could get anything from this weighted sum with with the weights and inputs, right? They could be any real number. That's all this is saying is

[3:00:00] like you get kind of an infinite range of possibilities. It's not really limited to anything. um which is fine for some problems like a regression for some problems like a regression that's okay but that is very open-ended

[3:00:14] like like if we want to generate a probability this is really bad because we don't want things to be really large or really big we actually want it to be confined to be between zero zero or one right coming out of the kind of final

[3:00:28] right coming out of the kind of final neuron so having an activation helps us kind of bound that output coming out of a neuron. Because if you think about it, this weighted sum could be anything. And actually, even in a regression, it's

[3:00:43] anything negative or positive. Like, if you're predicting a price, you probably want that to be positive. So, even minus infinity infinity is probably not even good enough. You probably want it to be restricted to between zero and infinity

[3:00:57] at the very least. So, that's where the activation comes in. we apply a function to this and that helps us kind of map that to a potential range like maybe we map it to 0 to infinity. Um so we so for like a

[3:01:14] for things like price temperature or whatever. Um or maybe we map that to uh 0 to one for like a probability um that that would be good for that case. So it

[3:01:28] just depends. It just depends on what problem we're solving. It also depends where we are in the network. Usually for things towards the end, we want to make sure we're using an activation that lines up the output to be in the

[3:01:40] producing probabilities, want that to be between 0 and one, not, you know, all positives, which just the weighted sometimes inputs could be anything, right? Could be anything. Um, and again, that's where the activation function

[3:01:56] this and allows us to kind of map this uh to to get a certain output from the output range at least for that neuron. Okay,

[3:02:09] so hopey that makes sense. So for that reason, activations play a really critical role in in any neural network. And as I said, especially towards the end of the network, you know, the example I always go back to is if we're

[3:02:23] predicting like three different things like um dog, cat, and giraffe, we don't place. We really want them to be probabilities that add up to one actually. So across this kind of out, let's imagine this was our output layer.

[3:02:38] We would want to make sure that these are all probabilities that are between like uh zero. So probabilities in 0 to one in this range.

[3:02:51] So we'd certainly want that to happen. We'd also want all of these to total up to one. We don't want them to be normalized in some sense, right? So um each one of these outputs should um total

[3:03:04] total across the output happen if there if there are probabilities. So that's something we're

[3:03:16] what we would hope is there's an activation we can apply across this activation we can apply across this entire layer really such that um all of these uh are probabilities certainly in 01 and total up to one for those kind of

[3:03:31] cases. So we're going to see activations that actually will do that for us be able to normalize that for us which is good. All right. Okay. So, and the other thing about the um activation function as well is the the activation function

[3:03:47] actually also serves a really important part of the model um because it allow it basically allows us to introduce um nonlinearity because if you think about it like if we just had let me go back a couple slides

[3:04:02] to this. If we just had this, this is basically a linear model, right? Just we've seen before. It's just a line, this is just a linear regression.

[3:04:14] actually all that that is. It's just weights times inputs plus a bias. We've seen that before, right? That's all that that is. So without activations,

[3:04:26] we essentially have just a huge collection of linear regressions that are um connected together. So activations actually play a really important role in introducing nonlinearity which is really really

[3:04:39] important for a neural network to be able to learn complex patterns. So activations are actually really critical. So it's having this F here, critical. So it's having this F here, this activation allows us to learn more

[3:04:53] general patterns than just linear ones because if we didn't have that, that's all that we would have is just linear relationships all over the place, right, throughout the network. Um, but having an activation function here that and the

[3:05:05] activation function will generally be nonlinear. It will gen generally be nonlinear. We're going to get into some examples but like some of them are going to be uh like the sigmoid like from logistic regression like a logistic

[3:05:19] function. Some of them will be hyperbolic tangent is a popular one. These are all you know highly nonlinear functions. There's also exponentials that are possible. So different types of functions that are all nonlinear. That's

[3:05:34] functions that are all nonlinear. That's actually a really big key uh because we want to introduce that in the learning process is the ability to kind of learn nonlinear relationships. That's another reason why they're important. They limit

[3:05:46] our range, but they also introduce nonlinearity so that our network can learn those nonlinear patterns, which is really important. Okay. All right. So, popular activations. Now, by no means is this going to be um an ex exhaustive

[3:06:02] list. There's definitely more than what's just on this list, but I'm going going to be the most popular that we deal with in this course for sure. You know, there are other ones that are out there that are more advanced. Um, but I

[3:06:15] would say they rarely show up for the use cases that we care about. The ones the ones that we care about are mostly going to be what's on this list. Um, and I'm going to explain what all of these are and kind of what they look like. The

[3:06:28] thing that these all have in common, by the way, is they are all nonlinear functions. All of them. So that's really that's a really important attribute of And again, we're going to have these spread out, you know, across all the

[3:06:44] spread out, you know, across all the neurons. And um so so in that way, we're these nonlinear relationships, which is which is um how we learn complex patterns. But let me go through these. So, these are some popular ones. We've

[3:06:59] little bit, but let I'm going to go we're going to start with that one and then we'll talk about these other three, the sigmoid, relu, and softmax. We'll popular activations. We've actually already looked at step, that's the one

[3:07:12] that's involved in the perceptron. Um, let's start there. So, the the Um, let's start there. So, the the reason this is called a step function is because it essentially only it has one step up from a zero to a one. So if you

[3:07:26] look at it on a graph, which I'll do on the next slide, it's a function that um goes from zero up to one and it's just a single step up uh one unit from 0 to one. Now this activation is kind of special to the perceptron algorithm

[3:07:41] because um that's just the way it's designed, right? Is that uh whenever that weighted sum of inputs that is being activated, right? that that data that's that weighted sum going into this function whenever that is um positive um

[3:07:56] we get a one as our result um otherwise we get a zero whenever that weighted sum we get a zero whenever that weighted sum is negative or zero so we we get zero so in that way it's it's uh let me show you it's kind of a step function that looks

[3:08:11] it's kind of a step function that looks like this so we this is inspired by activating a neuron right so if you have a single neuron it gets kind activated when there's enough energy. In other words, it's bigger than zero. And this

[3:08:24] is why it's called a step function is you can see on the graph, it literally looks like a step like like on the stairs, right? It goes up and then over. So, what this graph represents is, you know, this axis is the input

[3:08:40] and the y-axis is the output of course. And what you're looking at is for anytime the input is negative, so you're on this side of the x-axis. Um, you get

[3:08:52] zero as your output, right? You're you're down here, you always get zero, no matter what that is. The moment you have something positive, meaning you're on this side, um, you get a one, right? You get a one for all those, no matter

[3:09:05] no matter what that is. As soon as it's positive, you get a one. So, you get positive, you get a one. So, you get that step up to one. um and and that p that neuron is essentially activated, right? And we get a one as a as a

[3:09:17] result. Um so so this activation function kind of looks like this. Now function kind of looks like this. Now that is nonlinear for sure. Um but what we also see is the range of this is very limited to 0 to one. So the this step

[3:09:31] function activation is really only useful for this perceptron classification that's going to be zero or a one, right? really only useful for that because we're not able to produce any other value in between, right? We

[3:09:46] only get a zero or one. We also like for regression, we'd be out of luck because definitely wouldn't be a good fit for any of those kind of problems, right? So that's the step function. Again, this

[3:09:58] this one's only going to be used for the traditional perceptron. That's the only time it's really ever used is in that case. So let's talk about the sigmoid. This is one that we have seen before and it's actually the same function that we

[3:10:13] have seen with logistic regression. Um exactly the same function with logistic exactly the same function with logistic regression. So the sigmoid produces um a regression. So the sigmoid produces um a probability between 0 to one which also

[3:10:28] classifications. Just like the step function this is helpful for binary sigmoid um allows us to produce a probability

[3:10:41] that could be anything actually between 0ero to one. Right? So in that way we're actually generating a more direct probability rather than just a uh just either a zero or a one final result which we know from logistic regression

[3:10:55] a probability and what we did is we said okay if that probability is big enough meaning it's above.5 we categorize that as a one if it was down here we categorized it as a zero so we did that but the sigmoid activation

[3:11:12] allows us to produce a zero a probability between 0 and one as the output of a neuron. So remember, we're still applying this function to a weighted sum of inputs plus a bias maybe. And um now

[3:11:28] of inputs plus a bias maybe. And um now this function is the sigmoid is what we're saying is called the sigmoid. Um and this is going to map whatever this is um to between 0 to one.

[3:11:42] as kind of a probability between 0 to one. Um so where is this useful? This activation function is mostly used for binary classification as the output

[3:11:55] binary classification as the output neuron. So if you're in the network and so let's say let's draw the network. So let's say we have this network and we have maybe we have a hidden layer here. Um and then we kind of produce one final

[3:12:08] output. What we would, if we're doing uh binary classification like spam not spam or cancer not cancer, what we would really like this activation to be is a sigmoid. This this guy should probably be a sigmoid because what what we want

[3:12:26] to do what we want to do is basically produce a probability from this weighted Right? So we're going to take this weighted sum and uh generate a probability out of

[3:12:42] and uh generate a probability out of that. Um so we want this final result to be a sigmoid. Now these these others in the network don't have to be sigmoid activation um to produce their output. Um that's fine. We're going to look at

[3:12:56] for kind of the middle of the network that produces a general value. But um sigmoid is really really often seen at the very end when you want to produce a zero or one. Generally you don't want to

[3:13:10] produce a zero or one kind of in the middle cuz that limits your range, right? That limits cuz like then this this neuron would be very limited to a probability. This neuron would be kind of limited. This would be limited. We

[3:13:23] don't really want to do that in the hidden layers of the network. We don't we're not going to learn very effectively. Um but as a final output for a binary classification problem, yes, we want that to be a probability

[3:13:38] between 0 and one. So sigmoid works great. Remember the the function for a sigmoid is what we saw in logist logistic regression. It's this um fraction that looks like this. So it's it's uh

[3:13:53] definitely nonlinear, right? Definitely nonlinear. You can see it from the graph. But certainly, you know, this produces uh this produces um values between 0 and one. When x gets really big negative,

[3:14:08] um this fraction in the bottom kind of grows exponentially big. When x goes this way, it grows the fraction grows exponentially big in the bottom and makes it one over something really huge. Um which is basically zero. So that's

[3:14:23] why it decreases down this way as the x gets more and more negative. And then kind of the other way around is like if x gets really big positive and you go this way um you get closer to one because this term basically

[3:14:37] one because this term basically decreases down to zero um the larger x gets. So you get one over one. So you get something closer to one. So based on that function. But that's that's what the activation is. And and

[3:14:50] some this is called the sigmoid. Sometimes you'll hear it called the logistic function logistic function.

[3:15:02] Um it's the again it's the same one that's kind of used in logistic regression. uh in logistic regression remember this uh in logistic regression remember this it's basically this but we have um we

[3:15:15] have uh we have something like this beta we had our beta iixi we were doing that um so we essentially have that same thing uh for this activating that that neuron okay any questions about the sigmoid so when you use it is generally

[3:15:32] the last neuron in a binary classification if you're building a network for binary classification. It should be the activation on the final neuron. That's what it should be to produce a probability between 0 and one.

[3:15:45] We generally won't use it in the interior of uh like the earlier neurons because we don't want to limit the output of those earlier neurons prematurely, right? That that will that will basically collapse the learning and

[3:16:00] it won't be it won't be as effective. But we do like if we're doing binary classification, we do want to produce a probability as kind of that final output of the network. So that's why you'll usually see it as the last activation

[3:16:13] here. Okay. So sigmoid will sometimes see. Now let me show you the most see. Now let me show you the most popular uh activation. And the reason this is the most popular is because it's the one that's used almost exclusively

[3:16:28] in the hidden layers of a network. The reason is is because it does not really reason is is because it does not really restrict the values. In fact, what this this uh activation will just basically return back to you the input assuming

[3:16:45] that it's positive. So as long as it's positive, it's just going to give you back what you put in. So see how this function gives you the maximum between zero and x. So what that really means is like as long as x is positive, what's

[3:16:58] the max going to be between zero and anything positive? It's going to be whatever that other what that positive number is, right? So this will always just return to you what you put in as long as it's positive. However, this is

[3:17:13] long as it's positive. However, this is a nonlinear function because it has this piece right here, which is really critical. This piece is saying like whenever we have something negative, it's actually going to chop that off and

[3:17:26] just give us zero. So this is actually discouraging negative outputs. It's always going to give us something that's either zero or positive. Right? So this

[3:17:38] this activation function is known as a rectified linear unit. rectified linear unit. um because this y= x is a perfect linear um because this y= x is a perfect linear line um with a slope of 1, but we're

[3:17:52] rectifying that with um this kind of nonlinear piece, which is this uh flattening of anything negative to zero. Um because a true line would continue would still it would just that would be the true line, right? It would be um

[3:18:08] anything negative would just give us that value back in return. But we're actually chopping that off in favor of making it just zero. So we rectify it that way. The the way you pronounce this is called relu. So relu re lu. Usually

[3:18:26] how you'll hear that pronounced relu. Th this is this activation is the one this is this activation is the one that's most often used in the interior of that network. those hidden layers um because it allows us to basically

[3:18:41] because it allows us to basically generate the outputs that we care about um as long as they're positive. So it it doesn't limit us down to a specific range like 0 or one. It does limit us to

[3:18:54] a bit wider of a range than that like 0 to infinity essentially but in this case like these guys would likely be all relu activated. Um, and then like this one activated. Um, and then like this one would probably be a sigmoid for uh a

[3:19:08] binary classification. Doesn't choosing the max of the range skew the model in terms of prediction. We're not really choosing the max. Uh we're choosing the minimum uh because nothing can be negative. So it's always going to be

[3:19:22] negative. So it's always going to be capped at zero. Yeah, it so in theory, yes, I think that's a really good question. But in reality, no, because we're going to have many of these neurons um typically in the

[3:19:38] hundreds of these per layer. And basically what we're going to do is adjust the weights enough to make up for that. So remember, there's a bunch of weights here. And we're just going to make enough weight adjustments so that

[3:19:52] we can always produce something that carries forward into the next layer. So, um, the the weights, hopefully that makes sense, like the weights really help us learn because we're going to adjust those weights accordingly to

[3:20:06] produce the right outputs. That's actually all of what training a neural network is, is making the right weight adjustments. Um, and that's what we're going to talk about coming up soon. So, not really. We just have to adjust the

[3:20:19] weights appropriately uh to to generate the right outputs. And that's something that gets learned during the training process. So we'll talk about that. Okay. But this relu is um probably the most popular activation. We're going to use

[3:20:35] it quite a bit. It's used all over the place uh in most networks. Um relu is uh very very popular activation. It's it's kind of a very simple nonlinear function

[3:20:47] if you think about it. It's because it's almost always linear except in this case. This makes it nonlinear, right? Is this piece of it kind of makes it nonlinear, but otherwise it's mostly linear. Now, what

[3:21:02] a good thing about this activation as well is um like if we were producing a regression value for like a price, this would be a good one to have at the end, right? Cuz you can generate anything that's not negative. So, this would be a

[3:21:17] good regression candidate to use as an activation sort of towards the end, you're getting something that's positive. So that that would be like realistic. Okay. So this is a popular activation. We're going to use it quite

[3:21:30] a bit. And of course like in our code we'll be able to use this as one of our activations. And like I said where you'll mostly when you would use this you'll mostly when you would use this one is pretty much in any hidden layer.

[3:21:44] Your default is going to be relu. There's very few reasons you should not use a relu in your hidden layers unless you have a good reason not to. Um most like 90% of the time you're going to see relu used as as a activation on the

[3:22:00] hidden layers. The output layer is where you'll see those differences. You'll you can see a sigmoid you could see that step function in the perceptron and then softmax which will which will be used for probabilities as well. But most of

[3:22:14] the time we'll see relu show up in those hidden layers um between the input and the output. ReLU is very very popular. It's just an it's a very effective activation. It's a very effective nonlinear function that kind of just

[3:22:29] keeps things moving forward um as long as they're positive. Okay. So what's the difference between the sigmoid and the relu that we've looked at so far? Well, relu that we've looked at so far? Well, the sigmoid has a uh it's a little bit

[3:22:42] more complex of a function. It's that exponential that logistic. And one of exponential that logistic. And one of the issues with the sigmoid is it can be susceptible to what's called a vanishing gradient, which we'll talk about later.

[3:22:54] Essentially, this is this is a consequence of the sigmoid always being between 0 to one. So, the issue with that is it's a fraction, right? A sigmoid is going to produce a fraction that's between 0 to one. The issue with

[3:23:09] that is when we're doing the gradient descent learning process, if we're carrying around a lot of fractions, this is why you'll never see sigmoids in those interior layers is because if you're carrying a lot of a lot of

[3:23:22] fractions, they they multiply together with the because they become the inputs together with the weights of the next layer and really decrease that and and really limit your learning ability because you have such small fractional

[3:23:38] values. Whereas relu can avoid that because it is generally just going to to have a wider range, right? It's not going to limit you to 01. So it but it's

[3:23:50] learn those kind of complex relationships still with the help of kind of the weights and the training process. So, this is just to say you should generally be using a sigmoid in your output if you're producing a a

[3:24:06] binary classification output. You should avoid using sigmoids kind of in the hidden layers if you can because it's going to make learning more challenging. It's going to make training the neural net more challenging. And we'll study a

[3:24:18] little bit more the details of why that's the case, but it's mostly due to its range. It's producing the sigmoid produces fractions which carry through the throughout the network and it it kind of compounds like if everything is

[3:24:31] a fraction, things are going to compound really quickly and and make small values that are very difficult to learn from. Okay, so that's why you typically see relu in those uh hidden layers um because it makes sense. You don't want

[3:24:45] all over the place. Does that make sense? Like if I go I actually let me just illustrate that real quick. Is like you can imagine if we had uh going into

[3:24:57] a neuron if all of these guys were fractions coming out of a sigmoid from from other neurons in the layer. Um that's just going to compound, right? Like if this is if this is 1/3 and then this is like uh 1/2 and then you have um

[3:25:14] maybe another neuron that's connecting to it that has like a 1/4 then you just have a bunch of fractions that are being weighted. Um and uh they they're always going to reduce these values and and limit your range and kind of compound

[3:25:30] limit your range and kind of compound things. Um so so you don't usually see those sigmoids in the hidden layers. You'll see them at the end if you want this to particularly be a fraction. You'll see that. Okay. So relu

[3:25:44] nonlinear. It's just the maximum between zero and x whatever is going into that activation. And you'll typically see these in the hidden layers uh in between these in the hidden layers uh in between the input and output.

[3:25:59] one more activation because it's also going to be really popular uh especially it it's really popular especially as the output um using it in the output layer

[3:26:11] output um using it in the output layer for multiclass classification. So the softmax uh actually just extends the sigmoid. It's like a multi-dimensional sigmoid essentially. So the softmax produces values between 0 and one just

[3:26:26] probabilities. The difference is that the softmax um is useful for generating multiple probabilities that add up to one. So um what you'll typically see the

[3:26:40] softmax used for is the activation across the uh final layer in the output if it has multiple neurons in that final layer. So let's say let's say we had um

[3:26:53] three. So we we have a neural network that kind of looks like this. And then that kind of looks like this. And then we just have um maybe a hidden layer of four. And then our final output is three because we're we're predicting um dog,

[3:27:07] cat, and giraffe, right? Those three animals. Um what we would what we would use for this activation in this layer is the softmax. We would use the softmax for each one of these. And what that's going to do is produce probabilities

[3:27:24] that such that all of these are between zero to one and they actually are normalized such that they total up to one. Um, so softmax is really really great as you'll see it in kind of your final layer. If you're if you're doing

[3:27:39] multiclass classification, right? So, so you're predicting amongst a thousand things, then you'll have a thousand probabilities that total up to one. In this case, we only have three. So, we have three probabilities all that are

[3:27:52] between bounded between 0 and one that total up to one. Um, so you'll typically see this in the output layer. Um especially for like image classification um you'll see that or it could be you know multiclass classification

[3:28:08] uh you'll see you'll see this softmax another place you see softmax is in um another place you see softmax is in um like an LLM uh like a a large language layer because it needs to produce probabilities across every token that's

[3:28:23] probabilities across every token that's that's possible right so and the reason is is because the model like a GBT is going to predict what is the most likely next token and so that'll be represented by one of these probabilities right

[3:28:37] what's the highest probability as kind of the most likely next token so softmax incredibly useful for anything that's multiclass classification and it should be your last it should be in the output layer

[3:28:51] right it's going to be the activation that's used on the output neurons and the final layer to normalize them between 01 on and and get them to add up between 01 on and and get them to add up to one more importantly. So, um you

[3:29:05] know, it's a it's a really uh it's a really critical function. The softmax function, if you're curious what it is mathematically, is sort of like the mathematically, is sort of like the sigmoid except it's um more of it's more

[3:29:17] of an exponential. So, it's kind of like across a summation of these. Um so you uh you actually apply the softmax to individual components and then you sum

[3:29:29] individual components and then you sum that up over all the components. Um and so this normalizes it to be between 01 and uh make sure that the total of it is one um effectively. So it's a function that looks kind of like this and you

[3:29:44] apply it to a component in the layer. So every neuron in this layer is going to get uh this function applied to it. Okay, so S so just to recap that sigmoid it should be your activation on your final neuron in a binary classification

[3:30:02] um relu will be in your hidden layers in the middle it's the maximum between zero and x usually just returns x as long as it's positive and then softmax is going to be in your final layer for multiclass classification to produce probabilities

[3:30:17] that total up to one and by the way these are all all these activations are built into TensorFlow and PyTorch. So, we'll be able to use them really easily inside of our code. Um they'll be built into those libraries. So, uh it'll

[3:30:32] be really straightforward when we declare a layer of neurons. What activations do we want to use? Um it'll be really straightforward to set that all up when we start building our own networks. So all of that will be readily

[3:30:44] available and there's really good documentation in both of those libraries on kind of what activations are available. Like I said, there's more than this. You know, there are other ones like some some popular ones are are

[3:30:56] like hyperbolic tangent is a popular one. Uh especially with some image [snorts] be used hyperbolic tangent. Um that's a popular one. Um there's other there's variations on uh on relu. There's

[3:31:13] variations on uh on relu. There's there's one called jellu. Um there's there's one called jellu. Um there's there's one called leaky relu. them, but um the ones that we've covered on this list are the ones that we'll

[3:31:26] primarily stick to uh as we go through our examples. Okay, so that's a little that there are you know definitely multiple of them and uh which one we use depends on really the situation. All right. So I wanted to do a little bit of

[3:31:41] right. So I wanted to do a little bit of a demo here to show you an example with some activations uh and kind of how the activation is applied. Let me jump over to that. Okay. So we're going to look at this notebook. Do you guys have this

[3:31:53] one? This is the 3.06 uh taking a look at neural networks and activations. Hopefully you have this one. you want to follow along. Okay, one. you want to follow along. Okay, great.

[3:32:08] so what we're going to do the purpose of um Thank you for Yeah, thank you for sharing that. The purpose of doing this demo is uh mainly to see how we can build essentially a simple um neural network from scratch. So, not using

[3:32:24] TensorFlow or PyTorch, but see how we can use like a weighted sum through an activation and see that actually work in code, which I think will be really interesting to see. Um, and and really start to put the pieces together of a

[3:32:38] what a neural network looks like. Um, so we can kind of get a feel for the different components of it without yet going into our uh frameworks that are going to handle that more in depth like TensorFlow and PyTorch.

[3:32:53] So let's see. So first of all, we're going to import some things from NumPy. So some functions we might need like the dotproduct um random value um because we're basically we're going to build our own neural net that has

[3:33:07] those weights. So we're going to start out with random weights. So we need some randomization here and uh we'll have some various like exponential function and dotproduct things that we may need uh from numpy to do this.

[3:33:23] uh from numpy to do this. Okay. So what we're going to do is um essentially we're going to start by building a neural network uh a fake know, it's not going to use our frameworks that we're going to learn

[3:33:35] about, but it's going to show you the the basically from scratch some of the components that you would see in a typical neural network, including the And what we're going to do is basically build a perceptron neural network. So

[3:33:51] it's going to it's going to take uh three inputs essentially um an input that's a size three vector basically a 3x1 matrix and uh it's going to map it's We're actually going to use the hyperbolic tangent activation and and

[3:34:07] produce an output um which which would be kind of like a regression almost but just to show you the different components of this network to so you can see what's really involved with it. Okay. So to show you what's going on

[3:34:21] to build our kind of network from the ground up which is going to be kind of like a perceptron. What you're going to see is we need some weights and we have uh these weights which are generated randomly. Um and this ensures

[3:34:39] that their range is between minus1 and one because this uh numpy.random.random random generates a random value between 0 to one. Okay, so if we multiply something between 0 to 1 and multiply the by two uh and subtract one, kind of

[3:34:57] the range that we limit ourselves to is between minus1 to one in terms of those weights. So these are just going to be randomized weights that that live going to be random though. And the reason we're doing that is just to start

[3:35:10] out with randomized weights. This is actually how a network usually works is when you initialize the network, it has random weights to start with. What needs to happen is you have to train your network. So it learns those weights

[3:35:23] during the course of the training process. But this is pretty standard. Uh this is pretty standard that this is going to be randomized to begin with. networks in in our frameworks like PyTorch, TensorFlow, um those will also

[3:35:38] they get trained, right? How is this plural? Um because there's three of them. This random uh 31 generates kind of a a vector of size three. Um

[3:35:50] technically it's a matrix that's three rows and one column. So there's multiple weights. Yeah, that's that's why it's synaptic weights. Um because there's three of them that are just kind of in this like

[3:36:04] 3x one matrix which is just basically a vector, right? vector, right? with with three weights. [snorts]

[3:36:16] All right. So then we have our uh activation function here which is going to be hyperbolic tangent. Now all we're doing is when we give an input to this hyperbolic tangent. We're just going to um apply the numpy hyperbolic tangent to

[3:36:32] that input. Now what this is actually going to be our activation. Now we could use any activation we want. This is just an example. We could use relu. Um we could use uh sigmoid. This is just an example of declaring an activation and

[3:36:47] show you what that looks like. But you know if it was relu then that would be um you know it'd be like max between zero and x would be the relu. Um but in

[3:37:00] zero and x would be the relu. Um but in this case we're doing uh mp uh tanch which is hyperbolic tangent uh applied to x. Okay, that's just a function, just to x. Okay, that's just a function, just an activation function.

[3:37:17] which is um I'm not too concerned with right now because we haven't learned we're going to in in the later notes in this lesson. We haven't quite learned this lesson. We haven't quite learned it, is the gradient descent process for

[3:37:30] to learn all about that coming up shortly. But one of the things you need in order to uh in order to do gradient descent is the gradient of the activation um which is this uh so the derivative or

[3:37:46] the gradient of uh the hyperbolic tangent uh is this function here. That's just a mathematical uh function that involves that hyperbolic tangent. So the

[3:37:58] involves that hyperbolic tangent. So the reason we need this is the reason we need this is for training uh gradient descent. Um and again I'm going to go into detail

[3:38:10] on what gradient descent is coming up shortly uh in the notes but um it's just here in this example for the an example of how you would train this thing. Okay. So we have the activation derivative which will be useful uh in the training

[3:38:25] for gradient descent. Okay. And then we have a uh function called train. So this function is called train and takes the training inputs um training outputs

[3:38:37] which are kind of like the labels and then um a number of training iterations. then um a number of training iterations. Again this here is going to be our this is going to be gradient descent training uh process which we

[3:38:52] haven't learned about yet. So I'm not too concerned with us digesting all of this this moment, but just so we kind of preview what's to come. Essentially what we're doing here is an iterative process through our through through our data. So

[3:39:08] we basically iterate a certain number of times which is configurable this number of training iterations. We basically do this uh we iterate this many number of this uh we iterate this many number of times through this descent process which

[3:39:21] times through this descent process which is to say we basically produce the is to say we basically produce the output given our inputs. So we do a forward propagation. Um so that's what this is is we do a forward propagation

[3:39:43] that? It's because we want to see how far off our network currently is. So if far off our network currently is. So if we produce outputs, what we can do is actually compare that to our labels, which are these guys, and we can compare

[3:39:56] our output to that and see how far off we truly are. Um, which is what's called our error or sometimes called the loss. So this part here is um computing the

[3:40:09] So this part here is um computing the loss which is how bad our network is currently. So we're we're just gathering up how bad we are currently. Now why do up how bad we are currently. Now why do we do that? It's because we use that

[3:40:23] quantity to make an adjustment to our weights. That's the key thing is we actually want to make an adjustment to our weights to basically shift our network to perform better. Our goal is to minimize this error. I should I

[3:40:39] should write that down. The goal is to minimize this error. So essentially what we're doing is measuring our current error and then using that quantity along with the derivative of the uh activation which is

[3:40:57] here the derivative of the activation. We basically use that amount multiplied by that derivative. Uh we basically use this amount to figure out how much we should adjust our weight. So this is a numerical calculation that's driven by

[3:41:13] this gradient. Um so that that's why it's called gradient descent is we we essentially want to uh make an adjustment that uh will minim that will

[3:41:25] lower our error over the course of the iteration. So as we iterate a certain better and better and better at making predictions using our network. And you can see here like we make adjustments. So you see how

[3:41:38] we take our weight, we take whatever our weights currently are and we add in an adjustment. So, this makes a tweak. It think of it like turning a knob. So, we essentially turn the knob and the adjustment is telling us how much should

[3:41:51] we turn that knob. Should we turn it a lot? Should we turn it a little? We're figuring out how much we should make that adjustment to get better outputs of that adjustment to get better outputs of the network. So, this is um turning the

[3:42:03] knob so the network can predict better, right? So it so it can it can make the right adjustment and predict better in the future. So this in a nutshell is how any neural network is

[3:42:19] nutshell is how any neural network is going to train. Now this process going to train. Now this process will generally be handled by the framework. So by PyTorch or TensorFlow will manage this for us because it turns

[3:42:33] out when you have a pretty big network that computing this adjustment is incredibly intensive on a big network computing what that adjustment should be is is actually very computationally intense and it's called back

[3:42:48] propagation. So com so basically uh doing the weight adjustment and uh figuring out uh figuring out how much you should adjust and then actually turning all those knobs. You can imagine like for a very large network that's

[3:43:02] going to be an expensive operation. Just to give you some context like modern OLM uh have hundreds of billions of these guys, hundreds of billions of weights.

[3:43:14] So you need to compute an adjustment for every single one of them. hundreds of billions of weights, right? And you need and you actually need to do that every iteration. So you're imagine training for thousands of iterations and you have

[3:43:29] hundreds of billions of weights that you need to adjust. It's a that's why they take forever to train. [laughter] That's they're massive networks. They take forever. That's that's why uh because of this computation right here

[3:43:42] can be very expensive. uh because this derivative can be very uh computationally intense to compute. It on how big your network is. But generally those large networks like an

[3:43:55] LLM uh it's going to take a while. That's where you benefit from a GPU by the way is to try to make this computation faster. So we'll actually vision networks uh we'll do a similar training process and we'll utilize a GPU

[3:44:10] there to make this faster. Okay. By the way, I forgot to mention uh this forward propagation when we produced output uh we call it think mainly because we are uh it's like sending data through our network. So it's like all of our neurons

[3:44:26] are thinking together in this network. Um so thinking what is thinking? It's really um th this is uh doing our weighted sum through an activation that that's all it is. So it's like our our weighted sum through an activation.

[3:44:45] sum through an activation. So you can even see that here. Notice how we apply the activation, the tanch to this weighted sum. Now why is it a weighted sum? It's because you're taking the dotproduct between your inputs and your

[3:45:00] weights. So that literally now what does dotproduct do? Remember dotproduct adds together the product of these guys as vectors, right? So that's that's taking

[3:45:12] our weighted sum and then does that make sense? This is taking our weighted sum sense? This is taking our weighted sum and passing it through this activation. So that's this think is just computing that neuron essentially, right?

[3:45:25] Computing that neuron. All right. So just to finish out the demo, um that's the network. So that's our that's kind of our fake network. our that's kind of our fake network. It's got three weights. Um it it does uh

[3:45:37] it it thinks by doing a weighted sum of those inputs through so it has three inputs, three weights. Take that weighted sum, pass it through a tanch activation. Um and again the activation is kind of arbitrary. We could really

[3:45:52] could be a sigmoid if we were producing a probability here. it's just producing a tanch which is going to be tant has its um outputs are actually between minus1 and one. So we're always going to get something between there. So then we

[3:46:06] can run this code here which is going to take a random set of inputs um and then the labels for each one of those. So these are all like different labels like so we have four different input sequences. Notice how every input is

[3:46:20] size three which is what it should be. So here's here's one input. Here's one input. Here's one input. Here's one input. These are more or less random just just to show you how it works. And then here's a label for each one of

[3:46:33] these. So this guy is labeled zero. This this guy's labeled one. This guy's zero. So we have four labels here corresponding to each one of these uh sets of three inputs. So here's our random weights to start

[3:46:48] with is this vector. So we're printing that out just so we see what our random weights are. Now, when we go through the training process, we're just going to have 10,000 steps and use those random inputs and outputs. Look at what the

[3:47:02] weights are after we do this. The weights are now completely adjusted. weights are now completely adjusted. And we can actually put in a new uh input and see what the output would be. The output, which is just running think

[3:47:17] on this input, um it produces a label of one. Okay, so the network functions as we would expect. Um, and the big thing is to to take away here is to see how those weights actually get adjusted. You see how far off they are from what they

[3:47:32] started as. So here they started as these really random numbers between these really random numbers between minus1 and one. And uh here they get those weights actually get you know changed which is the objective of the

[3:47:47] training. Okay. Hopefully that makes sense. I think this is a good preview of what's to come in terms of the training process to come in terms of the training process and kind of uh how we um think about how

[3:48:01] and kind of uh how we um think about how this stuff actually learns. But you see about so far, things like an activation, things like a weighted sum going through the activation. It it's mostly just the components. I don't think you're not

[3:48:15] supposed to gleam anything from these outputs. uh these outputs aren't very meaningful because it's kind of a random input and it you know random activation. Uh so it's not a realistic example but the the point of it is to see kind of

[3:48:28] the inner workings of this network to see the different pieces of it. Like typically what's going to be involved in building a network is you know setting up an activations doing forward propagation backward

[3:48:41] propagation train it like there's training you kind of see what goes on in the training. It's mostly so we see that inner working prior to us building our own using using the frameworks mostly just so we get exposed to that. Uh okay.

[3:48:56] So let's talk about more let's talk more about those uh terms we just used mainly forward and backward propagation uh in our neural nets particularly um you know just thinking about it from a neuron perspective data going in um and then

[3:49:12] making those weight adjustments. We'll talk more about that. So training any neural net really involves two particular phases and we just saw those

[3:49:24] in the code really uh involved in that training function, right? And um those two phases have a name which is known as forward propagation and backward propagation. We're going to break down what each of those really means. Um, but

[3:49:38] it's kind of what the name suggests, uh, generally is that forward propagation is going to be all about sending data through to produce outputs. That's mainly what you want to do. And in fact, once you're done training and you're

[3:49:52] predictions, that's actually all you're doing is just putting data forward through it to produce an output, which is sometimes why that is called inference. Uh, because you're just making a prediction at that point. Now

[3:50:06] making a prediction at that point. Now the backward propagation is what makes those adjustments to all the weights across the whole network. And so when across the whole network. And so when you have a very um deep network, you

[3:50:18] need to be able to adjust all the weights across the whole network. And back propagation is going to be the algorithm that kind of does that and manages that weight update. And it's all going to be driven by this gradient

[3:50:30] descent that's that's dictating how much of an adjustment should we make to these weights. And then we basically need to propagate those adjustments throughout the network, which is going to be what back backwards propagation is actually

[3:50:43] going to do. And so you really only do backward propagation in the course of backward propagation in the course of training. But it is an iteration of doing forward propagation to generate predictions to see ultimately to see how

[3:50:57] far off you are in those predictions from an error perspective. Uh and then you basically do backward propagation to go back and uh adjust all your weights and then you just keep iterating doing these over and over and over until you

[3:51:11] make enough adjustments to kind of uh have a solid network that makes predictions really well. So training is really just an iteration of these two things. So we're going to we're going to dive deeper into what these two things

[3:51:25] are, but that's that's basically the summary of it. So I think forward understand of the two. And I think it's because it's really straightforward. It's essentially um it it's called forward propagation because when we have

[3:51:42] forward propagation because when we have this network of neurons um data flows kind of forward through all of the layers, right? It goes uh we have inputs that go into our first layer that generates some output which becomes the

[3:51:55] input to the next layer and on and on and on and eventually we reach that and on and eventually we reach that output. So in a perceptron in a which is a very basic obviously a very basic network basically just one neuron it's

[3:52:08] not even really a network but uh just a single neuron um we think of that as as forward propagation really as just producing the output and we know what does that what produces the output is the weighted sum through an activation

[3:52:23] right that's really what produces the output is this typical uh what we've been calling weighted sum uh through an activation and so this produces the activation and so this produces the output data flows forward in this

[3:52:37] process. So forward propagation is just generating the output through weighted when we have a more sophisticated network so if we were to draw multiple

[3:52:49] neurons in the input and then maybe some hidden layers um and maybe multiple hidden layers and then like a final output let's say and let's say that's a probability you know the the forward propagation is going to be a little bit

[3:53:03] have I can even draw this in a different color you know we're going to have going to be weighted and then contributions here,

[3:53:15] right? Contributions here, contributions here. So, it's this much more involved network. And then data is going to flow like this. And then data is going to flow like this.

[3:53:32] So, much more sophisticated um flow, moving this way, right? We're sending data in generating an output. That's the

[3:53:44] that's the forward propagation really producing that output. Okay. So for the either zero or one, right? Because the activation for the for the perceptron is this is basically this, right? So if that weighted sum is positive, we get an

[3:54:01] output of one. That's the activation that that gives us the final output. If that weighted sum is negative or zero, then we get zero, right? That is the that's just the activation function. Now, what needs to be learned are these

[3:54:19] weights and that's true of any neural net. All of those weights are really what need to be actually learned during the training process. So that's kind of what we saw in that in that demo is we need to make adjustments to those

[3:54:33] weights to basically have the best network structure that we can have. We want to make adjustments to those weights so that we accurately predict one or a zero or accurately predict the probability

[3:54:47] um for our labeled data. So we can't forget that training a neural net is problem, right? We still have labels like on our like think about images. We

[3:54:59] have all these images of cats, dogs, giraffes. Every one of those uh images giraffes. Every one of those uh images has a label associated with it so that we know if we're getting an error or not. And that can inform us making an

[3:55:12] adjustment to all of our weights. Um so weights which is what the training process will do. It will make adjustments to those based on the error that we're producing. We can only really

[3:55:26] forward propagation, right? Because we need to know what our network thinks is the right answer. That's again kind of in that demo we we saw it was called think is really the forward propagation.

[3:55:39] The weighted sum through an activation. That's what forward propagation does. It produces those outputs. Okay. [snorts] So we have forward propagation just the simple act of sending data through our network producing an output which we

[3:55:52] know is really just this series of weighted sum through activations even if we have a big network it's still going to be this collection of weighted sum through activations over and over and over again to produce some type of

[3:56:05] propagation. Now let's talk about the error function that we saw like in that error function that we saw like in that demo, right? So we need some way of evaluating where our network is currently. If we're

[3:56:19] producing results in order to make the right weight adjustments, we need to know how far off our network is from the true labels. And that's where something like a loss function comes into play. We've actually seen loss functions in

[3:56:34] machine learning quite a bit. Remember we had mean squared error, we had uh we had mean squared error, we had uh cross entropy um for classification like with logistic regression. Um so loss functions are not new to us but in the

[3:56:51] little bit new and we're going to break down kind of how they're used. But this entirely be new to us, right? We've seen it before in machine learning especially with things like linear regression. You guys remember we had the mean squared

[3:57:07] error loss function that told us how far off our regression was from from the the true value. It was kind of that distance, right? So in terms of neural networks, it's actually the same kind of concept. We have um we need some way to

[3:57:23] measure how far off our output that we get from forward propagation is from the true value. And this is really really important because this is going to inform how much of a weight adjustment we should make. Should we make a really

[3:57:38] big one? Should we make a tiny one? That's that uh that's that notion of turning the knob. Should we turn the knob a lot? Maybe just a little bit. That's all informed by how much error we have when we do forward propagation

[3:57:52] have when we do forward propagation compared to the true label. Okay. So, um that's just the general idea of the law. So uh in in neural network terminology um we usually think of this error we usually call it a loss function which is

[3:58:06] some type of function that computes what that error is between the label and what our network is producing. We usually call that a loss function and the goal of the training is to actually minimize

[3:58:20] that error minimize that loss function over the course of the training. But how much loss we have influences how much of a weight adjustment we make to the network. That's a really key idea in the training

[3:58:33] that that we should uh kind of understand intuitively that you know if we're really far off we need to make more of adjustment to our weights. If that far off, then it's a smaller adjustment we have to make to our

[3:58:46] weights. So it's it is proportional to that loss or that error. and we'll see exactly how proportional in in the gradient descent formula. Okay, so just gradient descent formula. Okay, so just to give you an example, you know, um it

[3:59:00] you know we've seen the mean squared error loss function before from something like linear regression, right? So we know like this value here would

[3:59:12] come from the prediction of the model uh in a linear regression that was our typical like beta times our times our features, right? beta 1 times feature 1, beta 2, feature two and we added those up and we could we generate a regression

[3:59:25] value and we see how far off we are from this in terms of the distance, right? That's the mean squared error. We've seen this loss function before. This is just an example of one uh that we see in in machine learning generally when it

[3:59:39] basically going to have the same kind of setup. We're going to have some type of setup. We're going to have some type of loss function we use that is supposed to loss function we use that is supposed to quantify this amount of error. So kind

[3:59:52] of what is our average error across our data? That's what this is getting at, right? Because this is averaging this is averaging our distances from all of our predictions to the labels. Um, and we want the same thing in the neural

[4:00:07] nets is to be able to average this amount of error. So we again can know how badly we're doing and use that to influence what type of update do I need

[4:00:19] to make to my network in terms of the weights. So the loss function by the way the loss function sometimes goes by different names. You'll mostly hear me call it a loss function but in in like literature or in other places you may

[4:00:33] hear it called uh cost function or objective function. um or sometimes we just call it the error. It's kind of all meaning the same thing. Um but I think the proper terminology is truly a loss function. And again, that's going to be

[4:00:49] be something that's quantifying that error. Uh and and there's different problems. So like this is a really good loss function for regression, right?

[4:01:01] Because we're measuring the distance between our prediction and what the true value is. that makes sense to do for regression. For something like a functions like in our logistic regression, right? We had the entropy,

[4:01:16] the binary cross entropy loss function which was uh a different different loss function that but it's meant to do the same thing is kind of penalize those mistakes, right? When you're making a mistake, you want that to show up in

[4:01:29] your error. So this function is a way of quantifying that. Okay, so we have a loss function uh and so you know we we see we've seen this picture visually for the mean squared error before uh you know the the

[4:01:46] difference between what our uh prediction and actual value uh is captured in that mean squared error metric. And obviously the greater that is that means on average the farther away our prediction is from the uh from

[4:02:01] the true value. So like in a linear regression the predictions are always on this line right? So the predictions are like here and here here they're always on this line and you

[4:02:13] can kind of compute this distance between this line and here. Okay. So why do we care about the cost function? um it's mainly so that we can do uh

[4:02:26] gradient descent. So I've used this word before but the great we're going to talk about that coming up in the next few slides. We mainly need that to to again compute how much of an adjustment we should make to our weights to make our

[4:02:40] model better. We can only do that if we know how bad we currently are. So if we go back to linear regression where we had a model that kind of looked like this to make our predictions that's actually how under the hood like in

[4:02:55] scikitlearn when we call model.fit fit. These betas were learned. Remember, these were betas times all of our features. These betas were learned in the bias were learned um actually through a gradient descent process which

[4:03:10] would uh find out what those are based on this error and uh kind of a a formula that takes into account that error to make an adjustment to these betas. um it's actually going to be the same exact thing for neural networks. It's just a a

[4:03:26] little bit more complicated of a formula to compute that adjustment because we have a more complicated model in the neural net than just a basic linear regression. Um but that's going to be the idea is we want to adjust our

[4:03:39] weights of our network and we're going to use the loss function to help facilitate that like how much of an adjustment we should make. So it's really really important in the training process of finding those weights of the

[4:03:54] network. Really important. Okay. So again like the gradient descent process in a linear regression sense would would basically find the best fit parameters those betas in order to minimize that error. We're going to do the same thing

[4:04:08] with the neural net. We're going to find the weights that minimize that error uh through a gradient descent algorithm. Any questions? uh on this picture are kind of uh let me slow down and say are there any questions about what the loss

[4:04:23] function is and kind of what its purpose is. Hopefully that makes sense. We're going to use it quite a bit as we go through setting up the training and executing the training of our own neural nets when we build them and and truly

[4:04:36] like the loss function will actually be something built into the framework. So they'll be built into uh TensorFlow and PyTorch. So you know we don't need to do it manually but it's good to have that intuition understanding of what they are

[4:04:49] doing and why we need it. So again it's capturing that error so that we can make adjustments to our weights and that's how we that's how the neural network how we that's how the neural network learns and makes itself better uh is by

[4:05:03] doing those weight adjustments. Okay. So let's talk about that weight adjustment. So so far what we have in the training process is forward propagation which sends data through and generates an output. What we can do with that output

[4:05:17] output. What we can do with that output is then measure through a loss function how much error are we producing because we have generated those outputs. So actually let me go back and and mention that like this um this prediction here

[4:05:35] uh I should call out is what we're producing from the forward prop right this is what we are actually producing in our network uh when we put data through is we're generating a prediction

[4:05:48] and so that's why forward propagation is is first it generates those outputs then we compare that to the true label through the loss function, right? And so this kind of tallies up and keeps track of how bad are we doing or what is our

[4:06:05] error in terms of our current network um setup. And again, the point is to use this quantity that we're measuring to influence weight adjustments to get a

[4:06:17] better network the next time in the next iteration. So let's talk about that weight adjustment. That's going to be the back propagation part of this. The forward prop is producing the outputs. We have a loss function measuring how

[4:06:30] labels. Now we need something to actually make the adjustments which is going to be the back propagation. So let's talk about that. So essentially let's talk about that. So essentially what the uh back propagation is going to

[4:06:44] what the uh back propagation is going to do is systematically make adjustments to all the weights in the network. So of course the deeper the network the more of take a look at this picture here. This is going to have a lot more weights

[4:06:57] involved in it than just a single perceptron which you know is just a single neuron and it only has a handful of weights depending on the inputs. Um but essentially what we need to do is make adjustments to these weights um

[4:07:12] make adjustments to these weights um that can uh make our network better. those weight adjustments. It's it's almost like it's turning all the knobs to configure itself so that it can

[4:07:24] effectively predict on on the data, right? It can effectively predict if this image is a cat, a dog or a giraffe. And so just to summarize how this is going to play out, the forward propagation produces the uh so the

[4:07:41] forward prop, let me just label that here. forward here. forward prop produces the uh output or the I know the output of the network. So it produces the prediction. The prediction

[4:07:54] goes into the loss function and we do that. We generate predictions against all of the data in our training set and and we can take our prediction set and and we can take our prediction we can take our labels as well and

[4:08:08] inject those into our loss function to figure out how how far off we are. That's the whole point. Now this loss function produces basically a weight

[4:08:20] function produces basically a weight adjustment which is what we saw in our code for that demo. Right? We have we're generating some type of weight adjustment through this error. And the

[4:08:35] weight adjustment is is uh a formula. It's a formula that's given by gradient It's a formula that's given by gradient descent. So it's uh it's actually a formula that's computed based on the gradient of this loss function and it's

[4:08:49] uh directly influences how much that adjustment should be. We're going to learn about that. But the back where back propagation comes into play is now back propagation comes into play is now that we know the adjustment, we need to

[4:09:02] weights. We need to systematically update all the weights. And it turns out that it's not trivial to do that bec and let me tell you why. It's because to update a weight that's earlier in the

[4:09:17] network. So let's say let's say we want to update this weight on this to update this weight on this connection. This weight here. Let's call it W. If I want to update that weight, the

[4:09:30] If I want to update that weight, the update there actually depends on the update there actually depends on the updates in the further layers down in the network. So the the adjustment here actually is dependent on the adjustments

[4:09:45] to all of these guys. So to everything after it, it's dependent on um in order to update this guy that's earlier. So for that reason, we actually work backwards. We work back to front which is why this is called backward

[4:10:00] propagation. We need to start here and figure out what are all the weight adjustments I need to make from here and then we can use that to then update all the weight adjustments in the next layer and then in the next layer and then in

[4:10:15] the next layer and then in the next layer. So this is why it's called back propagation or backwards prop. back prop back propagation

[4:10:31] and the reason there's a dependency there is all about calculus it's it's a very I don't want to get into the math of it too much but it's all about calculus essentially it's because the gradient

[4:10:43] of a weight earlier on in the network is dependent on gradients that follow it dependent on gradients that follow it later on in the network this way so um in order to compute earlier updates, we need to compute updates that are further

[4:10:58] need to compute updates that are further down the network. So algorithmically, it makes sense to start here on the end and then once you know those updates, you just work your way backwards and you can use those to compute the updates here.

[4:11:11] Then you use these guys to compute the updates here and it works its way backward. Okay, so this is this is actually the essence of training. You do forward propagation produce a prediction use that prediction to generate some

[4:11:26] use that prediction to generate some loss. That loss is comput is uh used to compute a weight adjustment for every weight. So the weight adjustment is a formula that's that's utilizing the derivative or the gradient of the loss

[4:11:39] with respect to every weight and you start with weights towards the end of start with weights towards the end of the network and then you back prop towards the front of the network to make all those adjustments. Ultimately you're

[4:11:52] going to adjust every weight in the network. Uh but you're just going to do it starting from back to front backward propagation. That's what that does. Okay, it it sounds complicated but in

[4:12:06] reality like we will not manage this process manually of course. So this is process manually of course. So this is the beauty of TensorFlow and PyTorch. This is why they exist because they manage this network propagation for us.

[4:12:23] They manage all of this for us. That's really why they were created was to make this so much easier of a process to manage the flow of data through this network whether it's forward or backward with the gradients and all that. It's

[4:12:38] complicated. I agree it is it is complex like training a neural net is not trivial. Uh it is a complex process but the um frameworks that we're going to learn with in Python the TensorFlow and PyTorch make this relatively easy. They

[4:12:53] they make this relatively easy but that's the beauty of them is they do make it easier on on us but I think it helps to understand these steps forward figure out how bad you're doing with the loss function compute your weight

[4:13:10] derivative of that loss that's the gradient and then you propagate that gradient and then you propagate that back uh from back to front because earlier weight adjustments ments depend on later weight adjustments. So you have

[4:13:26] it makes sense. You cannot do earlier ones without knowing the later ones. So it you really have to go back to front. You have no other choice because you and update these guys. You would in order to update this, you need to know

[4:13:40] this. In order to know this, you need to know this. In order to know this, you need to know this. So you have to start from the back and go to the front. You really have no other choice to do the adjustments. Okay? And again, that's why

[4:13:53] I said this stuff is not trivial. It's definitely advanced. Um, but really glad definitely advanced. Um, but really glad you guys are uh, you know, trying to it's well worth it to understand how to work with neural nets. As I said, one of

[4:14:07] the preeminent examples of neural nets are like generative AI. Neural nets are all over the place. So, it'll help to understand, you know, that how those cover those later on, things like

[4:14:20] transformers. Okay. So to recap that process, um you know what we're doing here is everything I just said. We're uh basically setting up our network with the initial weights. So when you first build your network, you're going to have

[4:14:36] random weights. And we saw that in our code in that demo. Everything has random weights. The goal is to learn what those weights should be over the course of weights should be over the course of training. So in order to um in order to

[4:14:49] propagation which is what uh this first step is. So this is forward prop step is. So this is forward prop because we are producing the output. Um calculate the output just means forward propagation right send your data

[4:15:04] through generate an output. Um then you do your update on the weights which is through uh back propagation right. go back prop because uh we're going to compute a weight adjustment. Now, this is saying

[4:15:19] that the weight adjustment that you should make, we're going to learn this a little more, the weight adjustment that you should make is based on the that you should make is based on the previous weight plus this amount here.

[4:15:32] And this this is just saying it's proportional to your error or your loss proportional to your error or your loss cuz this is your loss in a simple uh in a simple simplified way, right? It's saying here's what your label is, which

[4:15:47] is D. Here's what your predicted output is. Uh here's what the predicted output is. Uh here's what the predicted output is Y. And uh the gradient of this is typically proportional to the input X. And then there's also some learning rate

[4:16:01] that's involved. And this is this is just the gradient descent formula. That's all that this is. So we'll learn a little bit more about that coming up uh especially in lesson four or maybe in less the end of lesson three. Um but uh

[4:16:16] this is the gradient descent formula. All it's saying is this is how much I what it's proportional to. It's the error. It's really proportional to that error. Not only the error but the derivative of that error which again we

[4:16:30] don't have to compute manually. PyTorch or tensorflow would compute that for us. They will keep track of that and compute what that adjustment should be. But coming from this gradient descent algorithm which is basically saying in

[4:16:45] order to make an adjustment I need to take some proportion of my error in particular the derivative of the error. Okay. And then we iterate. So essentially we iterate these steps over and over again. So iterate these steps

[4:16:58] over and over again until you've completed all your steps which is basically convergence. Maybe you do 10 steps, maybe do 100, maybe a thousand. Um that's actually configurable is how many steps do you want to take and um

[4:17:13] take more, sometimes we need to take less. Depends on the situation. Um, but we continue to do that and every time we do it, we're adjusting these weights. And the thought is like the next time around these weights should be a little

[4:17:28] bit better. So our error should hopefully decrease. One of the things we should look for is the error should be steadily decreasing. That's what should happen, right? Because we're making an adjustment. As long as we're making a a

[4:17:41] solid adjustment, which we should be because we're doing gradient descent, we should be getting better. So the next time we do this and do forward prop and then measure our error, it should be better and we should have less error and

[4:17:55] therefore we need to make less of an adjustment. So over the course of the training, we should be getting better, right? And and hopefully by the time we've done all of our steps, we're really good. Our network is performing

[4:18:08] even better than it did at the beginning, obviously. What is T? Uh so T is not really a true variable. it t is just uh it's like a step. So it's like an iteration count. So all this is saying is like if I want to update the i

[4:18:22] saying is like if I want to update the i weight for the next step. So t+1 is just saying like on the next iteration um you just take the the weight at the previous iteration. So what it currently is t. So

[4:18:36] like time t plus this adjustment. That's all it's saying is you take you take all it's saying is you take you take what it currently is at this time and add this adjustment to it. So this is just saying in order to update the

[4:18:48] weight you take what it currently is and add this adjustment. So think think of t like a step an iteration. So the weight at iteration. So if you were to plug in like the beginning like t equals 0. This is saying the weight on the first iter.

[4:19:03] So like on the first iteration t equals 0. If I want to make an adjustment, um, the weight will now become what it was at when I started. So the weight at at

[4:19:15] at when I started. So the weight at at t0 plus this adjustment. And this this way, it's not always going to be that, but it depends on how complicated your network is. But that this generally is the formula. Yes. T is iteration. Yes.

[4:19:30] So the the by the way the back propagation is this step here um that I I wrote down. This is the back propagation. Um and and again that's it's called that because this formula

[4:19:44] cannot be applied for earlier weights unless I know the the value of um later unless I know the the value of um later weights because generally this is this weights because generally this is this formula will involve weight updates of

[4:19:59] of weights that are further down in the network. So for anything earlier, it's weights are in the in particular the derivatives of the weights for um later

[4:20:11] adjustments. That's just it's just a calculus fact that is basically known as the chain rule. So if you're curious more about that, you could look up um back propagation and chain rule and see uh if you want

[4:20:25] some more like math intuition behind that. But that that's the reason the chain rule from calculus why we have to do back propagation. So I wanted to you know take a look at what the error like a typical error function might look like

[4:20:39] a typical error function might look like in terms of uh the some of the square differences. So basically like the mean squared error uh over the course different weight adjustments. So obviously what we're looking for is to

[4:20:54] obviously what we're looking for is to minimize that that loss. And the goal is to make weight adjustments enough so we end up down here at these kind of minimum loss values. Um we would love to end up here or here. We would love to

[4:21:08] there. What makes this tricky is that um in terms of like a minimization, you can the the gradient descent can can kind of

[4:21:20] get tricked in some cases to believing it has found a minimum in these kind of it has found a minimum in these kind of valley spots. So like here it kind of looks like a minimum, right? Even here and certainly even here it looks like

[4:21:36] we've bottomed out in terms of the error. And so what can happen is you can think that you've bottomed out and you can basically get stuck. You can get stuck here in these error positions where your error you believe you've

[4:21:51] you've hit a minimum. And so um your weight adjustment is not going to be that much. you're basically going to be flat and and at like an equilibrium and you're not going to be able to make a weight adjustment and climb out of that

[4:22:04] uh to to climb out of it and hopefully find a better minimum. So like ideally you're finding a really low minimum of of the error. Um and that's what you want, but you can get unlucky in certain um situations where you can kind of get

[4:22:19] stuck. And we're actually going to talk about things that help prevent this. So there are um certain algorithms that are out there that have been around for a while that that basically propel you beyond these uh valleys and and help you

[4:22:36] search for the lowest minimum you can in the error. Um because what you don't want to happen is again you don't want to get stuck here. Um in which case you haven't truly minimized your error. you think you have because when you look

[4:22:49] around, this is the classic like local versus global problem is when you look around from this perspective, it looks like you've hit a valley. It looks like you've hit a minimum. Um because everything around you is increasing. So

[4:23:03] it looks like you've hit a a valley and you can't get any better. Um but truly there's, you know, more of a minimum down here or down here. So where you really want to end up is one of these guys. Um, and so the training, luckily

[4:23:19] the training algorithm that we're going to use is going to help propel us to these minimums of the of the loss and and help us avoid getting stuck in these and help us avoid getting stuck in these uh kind of fake minimums. Okay. So from

[4:23:34] a perceptron perspective uh computing the error is actually really easy if if we just take the difference because what you can see is that like if you imagine what our prediction is our prediction is going to be 0 or a one and uh the the

[4:23:49] label is going to be a zero or a one in terms of a binary classification. So we could always compare the uh we can compute the error relatively easily with just a simple difference. So we could say like, okay, here's our prediction um

[4:24:03] which is one and here's the actual label which is one. So we had no error. It's kind of like a true positive, right? We got that one right. So there's no error. Um but we here we predicted one and it was actually a zero. So that's an error.

[4:24:16] was actually a zero. So that's an error. So that's an error of one. Uh so so that contributes an error of one if we take the difference there. Now that's a very simple error function. In a lot of cases, um, we actually want something

[4:24:29] error function because in a lot of problems, we're not going to produce just a 01 will actually produce a probability. So that's where things like probability. So that's where things like the entropy will come into play um, in a

[4:24:43] little bit more advanced of an error error calculation. But you know you could think like you could think of this as one simple way to tally up that error which is kind of like you know predicting uh this is kind of like a

[4:24:55] false uh negative. So actually sorry this is like a false positive. We predicted one but actually was zero. That's an error. Okay. So this is just going through everything I said and once we know that error so that's an example

[4:25:09] error calculation. Of course, the true like loss functions that we we will use that. They'll typically be like uh maybe but in a classification it typically will be something else. The reason being

[4:25:23] is because those lend themselves better to working with probabilities, which is generally what we'll produce for most neural networks for classification is a probability rather than just a zero or a one. But once we compute that error, now

[4:25:39] this is talking all about back propagation, which is just the term we're giving to going back and traversing back to front and making all of our weight adjustments to all of all the weights throughout the network. And

[4:25:54] and ideally we um get better at this as we go along. adjustments and then we iterate, make adjustments, iterate, make adjustments, iterate and we should be getting better each time. Our loss should be reducing

[4:26:11] over the course of these iterations, right? So everything you know everything we've been saying just kind of summarized here. Okay, so the weight adjustments are according to formulas like this. Now these uh the these um

[4:26:25] formulas are coming from gradient descent. So they're they're not out of thin air. Um these formulas are relative to your loss function and really like the derivative of the loss function. Um so they're not always going to be this

[4:26:40] so they're not always going to be this simple. In fact, they're they're usually simple. In fact, they're they're usually um more complicated, but um generally the formula is something along the lines of your new weight

[4:26:53] of your new weight is equal to your old weight is equal to your old weight um plus some learning rate times the um plus some learning rate times the times the um gradient

[4:27:10] function. So this is this is more generally the formula. It's just when you're using a simple loss function um it it it turns out to just be this is the gradient just this quantity. But we're going to learn more generally that

[4:27:25] it's more of an update like this uh generally um is is the gradient descent. generally um is is the gradient descent. One thing to to realize is instead of making an adjustment 100% related to the gradient, there's

[4:27:41] actually a what's called a learning rate here, which is a parameter that we have control over. It's actually a hyperparameter. A and there what this hyperparameter. A and there what this thing is doing is preventing us from

[4:27:54] thing is doing is preventing us from taking way too large of a step. So when we make this adjustment, we don't want to make an adjustment that's completely governed by the gradient of the loss. I'll I'll tell you why. Like the

[4:28:09] this picture. Let me draw a picture to kind of explain why. And this actually basically explains gradient descent is this picture. So imagine imagine um this is our loss function, right? And we're trying to

[4:28:25] reach the minimum of the loss. So we're trying to we're trying to get down here and what we can evaluate on every iteration is where we currently are on this loss function, right? We know from forward propagation. We can compare that

[4:28:38] to our labels and we can compute where we currently are. Let's say it's right We know we're somewhere on this loss function graph. Let's say it's right function graph. Let's say it's right here. Our goal is to move towards the

[4:28:51] minimum. We want to make weight adjustments that move us towards the adjustments that move us towards the bottom. Right? Now, imagine, now let me turn this to an analogy of hiking. Imagine we were working our way down a

[4:29:03] that mountain. One of the things we want to avoid is taking a gigantic step, right? And we risk kind of like tripping and um maybe traveling way too far. you

[4:29:17] know, if we if we go too far in any one direction, what can happen? And if going back to this graph, um if we take too big of a step, we can actually um end up

[4:29:29] big of a step, we can actually um end up over here, which would be bad. So, the idea is imagine working our way down a mountain kind of in the dark. Let's take small steps so that we guarantee we don't overshoot. Cuz if we take huge

[4:29:45] steps, we might overshoot. You might come all the way over here and and overshoot. So let's take small proportions of that step. So that's what this that's what this learning rate is. It's a fraction that controls taking a

[4:30:01] small step towards the minimum. And by the way, this is why it's called descent is we want to make an adjustment to our weights so that we descend down this weights so that we descend down this loss function landscape and end up at

[4:30:15] the minimum of the loss function. But we don't want to take such a huge step that we just skip over the minimum entirely. So the learning rate controls

[4:30:28] how big of a step do we take and usually you want that to be not incredibly tiny but not too big because it the tin does it make sense that like the tinier it is down the mountain. It's going to take us forever if it's a really tiny uh

[4:30:44] proportion right we're going to take baby steps and barely make any progress. same time, we don't want to risk taking way too big of a step and then jumping over the minimum and then and then we're we haven't minimized the the loss. So

[4:31:01] gradient descent a basically is governing we should take a step towards the bottom. And by the way, what's the bottom? It's in a decreasing slope. So

[4:31:13] and that's the gradient. The gradient represents kind of the slope of this loss function. So, we want to take a step in the direction of the gradient of this loss. That's why it's called gradient descent. Okay? And again, we

[4:31:26] gradient descent. Okay? And again, we don't want to take way too big of a step cuz then we'll jump over that minimum. So, we're going to take gradual kind of steps governed by this uh learning rate. Um, by the way, a typical value of the

[4:31:39] Um, by the way, a typical value of the learning rate is something like 0.01 or 0.001. But, and that seems small, but it's really not. Um that's this has been proven to be a pretty good value uh for

[4:31:52] the learning rate for most most examples. This is these have been proven to be pretty good um fractions of the of the error um so that your your step is a meaningful step but it doesn't risk going way too big. Okay. So this is this

[4:32:07] is gradient descent in a nutshell is it's it's a gradient descent is an optimization procedure. It's an optimization algorithm that says in order to find our way towards the bottom of a loss function, which is what we

[4:32:20] want to do. We want to make our way towards the bottom of the error. In order to do that, let's let's take a step in the direction of the gradient of the loss. That's because the gradient is the slope of this guy. So, no matter

[4:32:35] step in the gradient direction because that's going to bring us towards the bottom of the mountain, right? And we want to control that step by some amount so we don't overshoot it. We want to take a controlled step in that

[4:32:49] direction. All right, I know this might be a little complex, but hopefully um that helps a little bit on gradient descent and kind of what it's doing and why the formula is like this. The formula is like this again because we're

[4:33:03] taking a step. We're we're updating our weight in the direction of the gradient. weight in the direction of the gradient. This here is just a it's um the gradient This here is just a it's um the gradient of the loss. That's all it is. More

[4:33:16] generally, it's the gradient of the loss function. So, we're taking how far off function. So, we're taking how far off we are and using kind of the rate of change of that to figure out what direction we should adjust our our

[4:33:28] weight or how much we should adjust our weight. Okay. So, I have a an example of back propagation um that I will go through. It's kind of an in-depth example. So what I think we'll do is

[4:33:42] take uh a little bit longer of a break now and then we'll come back and go through an example of back propagation just so you can see kind of the flow of the data going back through and see truly why we have to go back to front um

[4:33:57] adjustments uh according to that gradient descent formula. Um so we'll do that. Okay. So, I just wanted to walk through a back propagation example to through a back propagation example to mainly show you how um the weights get

[4:34:12] adjusted uh back to front um through a simple example. There's now as a disclaimer there's going to be quite a bit of math involved with it and I'm not too concerned with the math but more the

[4:34:25] process just so you can see how there's kind of a gradient descent update that's relies on weights later on in the network to update weights that are earlier in the network which really is what the back propagation is all about

[4:34:39] computing those later ones then using that to update those earlier ones. Uh but the goal of back propagation is to have all of our weights be updated according to our uh adjustment formula

[4:34:52] and so that uh that's how we learn right that's how the neural network learns and trains is it it makes those adjustments uh each iteration. So, as an example, uh uh each iteration. So, as an example, uh we just have some kind of fake values in

[4:35:06] here just just to uh put some numbers to things so we can see uh kind of a a typical like network setup. So, this is a this is a very small network of course. It only has a couple of inputs and then it has uh a few a couple of uh

[4:35:25] hidden layer neurons. So it just has these two neurons right here as in the hidden layer and then it produces two outputs um which are uh basically like probabilities essentially uh in the

[4:35:39] example of something that's kind of like softmax here in this output layer to produce um kind of two probabilities that add up to one. But what you can see in this network structure is there are weights that have these green values

[4:35:54] associated with them. And obviously in the beginning those are random. So there's really no um special reason why they're those numbers. Every network starts out with kind of random values

[4:36:07] there for those weights. And as we keep saying, the goal is to adjust those weights over and over until we really learn and make uh predictions effectively with with this network setup. And so the initial uh input

[4:36:23] setup. And so the initial uh input values that we have are uh 0.05 for values that we have are uh 0.05 for input one and 10 for input two. So th those are the two inputs that we have to start with. And of course we have all

[4:36:35] these other weights initialized to kind of random values that are going to be adjusted in the uh back propagation process. Um so this is this the setting

[4:36:47] just for this is just a random example. Okay. And the purpose of the example is to kind of walk through over the course of the next several slides how we would adjust all these weights. And and as I said before, one thing that I really

[4:37:01] want us to pay attention to is the fact that we cannot update this weight or this weight or this weight or this weight or this bias. We can't update any

[4:37:13] weight or this bias. We can't update any of these guys without knowing the update this, without knowing the update for this, and without knowing the update for this. These are required in order to update these based on the gradient

[4:37:29] calculation. They are it's required to know ones that are later down the line know ones that are later down the line in the network. So that's why back propagation goes this way because you need to know these in order to update

[4:37:43] these earlier ones and you're going to see that directly in the formulas that we're going to show in in the upcoming slides. Okay? So, that's something to be on the lookout for. Uh, again, there's going to be quite a bit of math. I

[4:37:56] but it's important to see the back propagation kind of in action in this propagation kind of in action in this example. Okay, so let's see uh let's see how we can um use back propagation to kind of make adjustments to these

[4:38:11] kind of make adjustments to these weights. Okay, so uh we have those two neurons we just said. We have biases, we have weights. They're all initialized here. And that's fine. So nothing that that uh

[4:38:23] And that's fine. So nothing that that uh special. And then we have uh two outputs uh which we see their current values of 0.01 and 0.99. So we have those two outputs. Suppose right suppose we generate those as the forward

[4:38:37] propagation outputs. Okay. So the forward propagation the the forward propagation would generate um would would generate the output of each neuron throughout the layers. So for hidden layer 1, let's take a look at hidden

[4:38:52] layer 1, let's take a look at hidden layers one's uh input. So if we go back to the screen, H1, let's just focus our attention on this. Let's let's look at forward propagating through the network, right?

[4:39:04] propagating through the network, right? H1 has contributions H1 has contributions from uh this from both inputs. It has from uh this from both inputs. It has weight one times input one, right? and

[4:39:16] weight one times input one, right? and it has um it has weight three times input two and it has a contribution from bias one. So what we should be doing in bias one. So what we should be doing in order to compute this neuron

[4:39:30] uh is we should be doing that weighted sum through an activation. Okay, let's go back. Let's take a look at that. So going forward this is our uh this is our computation, right? It's the weighted sum. Um the the

[4:39:47] weight one time input one, weight three, input two, and then bias uh technically times one, but really it's just plus the bias, right? So if we were to compute that, we get this value as the input. And what we should be doing for H1 is

[4:40:04] really doing an activation to this quantity, right? This is the quantity we would be passing into an activation. um whatever activation we had there. So um whatever activation we had there. So um for example, if that activation was

[4:40:19] relu, it would actually just leave this as it is because remember the relu is just the maximum between zero and this input. So uh even if we applied f as the

[4:40:32] relu, it's still just going to equal this quantity. So to so do we see how the relu would kind of function there on that weighted sum we get 3775 even if we applied rel to it would still

[4:40:47] be 3775 cuz that's that's what the maximum is it's just that value okay so that's for H1 and uh actually the so I guess it wasn't

[4:40:59] mentioned there but the the activation that they are actually using it looks that they are actually using it looks like is the sigmoid So um just as an relative looks like we're using the sigmoid I spoke too soon. So it's using

[4:41:13] the sigmoid as the activation. So uh here we get.59326 and um carrying out the same process for H2. So if I go back to the picture H2

[4:41:33] what contributes to H2? What contributes to H2 is weight two times input one and then weight four times input two and then plus the bias right so those things should contribute to this we should be doing that weighted sum and then passing

[4:41:48] that through the activation which is the sigmoid and that's how we produce those uh those quantities so that's how we produce these two values okay so so far just forward top. Nothing that interesting going on besides just a

[4:42:04] weighted sum through an activation. So we're just doing forward propagation at we're just doing forward propagation at this point. Okay. So then what we're going to do is the same exact process to

[4:42:16] compute the output neuron. So if you look at the output neuron, let me go I have to keep going back to this. We're going to compute the output one. So essentially what the uh output one is going to be is um taking the weighted

[4:42:31] sum right so we do um weighted sum of this guy from H1 input and then a weighted sum of this guy times H2's input and then plus this bias should all

[4:42:43] contribute to output one and we should be passing that through the activation which I think is going to be another uh sigmoid to generate output one so let's See? Yep. So there it's another sigmoid. So

[4:42:58] Yep. So there it's another sigmoid. So we have uh we have this here. So we we we have uh we have this here. So we we generate uh weight five plus weight 7 * this plus the bias and we generate uh this quantity and then we pass that

[4:43:11] through the activation and we produce this value for output 01. Now we do the exact same thing for output two which would be again going back to the picture. So for output two what contributes to that would be should be

[4:43:28] contributes to that would be should be this amount here times H2 and then it should be um this amount here times H1 and then plus bias 2 and then uh we pass that through the activation right so those are the contributions for H2 or so

[4:43:43] sorry for output 2 and we generate a similar we generate and we generate a similar we generate this output uh here.

[4:43:55] showing you an explicit example of forward propagation. Hopefully that makes sense. The weighted sum through the activations and these are what we get for those two neurons in the output. So that's what we have so far, right?

[4:44:08] That's what we have so far uh in our in our diagram. Um we have not made any forward propagation with the given inputs and uh just taking the weighted

[4:44:20] sums uh through through the activations. So we get these two values currently. So we get these two values currently. Right? Now let's compute the uh mean squared error. So we have labels on these guys that was the original values

[4:44:35] here. So these blue numbers here for 01 and O2 are the labels for what these two should be. Okay? These are the labels for what those two should be. Let's compute the error so that we can do back propagation and update all of our

[4:44:50] weights. So we should be able to do that and we're going to use a mean squared error to compute the error. So you know not that different than what we've done before with like uh linear regression

[4:45:04] for example. So what we can do is compute the two errors individually. So compute the two errors individually. So we have basically uh we have error one we have basically uh we have error one is target 01 minus uh output 01 squared.

[4:45:18] So we have this quantity which was the 0.01 minus this squared and we take a half of that which is this. And then error two is um you know we take we have

[4:45:31] um basically the same thing but for target O2 and output O2 we generate target O2 and output O2 we generate error O2. Um so we have both of those errors and we should be able to total those together to get the total error.

[4:45:45] Okay so this is just computing our loss value so far. So if we take the total of value so far. So if we take the total of that um we have a total error of 3038. So that's our total error. Okay. So error one was this amount. Error two was

[4:46:01] this amount as the mean squared error between this and 001 and then this and.99. Right? We were comparing those two values generating our error. Okay. So just computing the error. What we need

[4:46:15] to do is use this error with our back propagation and uh gradient descent propagation and uh gradient descent formula in order to make adjustments to all these weights now that we know what that error quantity is. We're going to

[4:46:30] according to the gradient descent formula. Okay. So, so again, this is where we do forward and now we're going to do backward to adjust all our weights now that we know what the error is. Okay. So we're we're going through this

[4:46:44] process step by step just so you can see it. Okay. So let's do our adjustment to it. Okay. So let's do our adjustment to do back propagation. So this is where uh we want to do our gradient descent formula. Oh there was a question sorry

[4:47:00] formula. Oh there was a question sorry how it calculated error 2. Oh so error 2 is from it is doing this formula again. So it's um 1/2 of the uh target

[4:47:14] So it's um 1/2 of the uh target of output 2 minus the actual output of of output 2 minus the actual output of um uh neuron O2 uh squared. So it's the mean squared error between those two guys. So remember um this had a value a

[4:47:31] label of 0.99 from the beginning and then this had a value of uh whatever it was.77 or I think it was 79 something like that and so we're computing this this uh mean squared error between the two and then

[4:47:46] squaring that does that make sense so we're gener it's the same formula yeah it's the same mean squared error it's just from like we it was 76 six. Sorry, it was 76 because we computed these from forward propagation. We know what those

[4:48:02] outputs should be and they originally had labels from the data. They originally had a like output one had 001, output 2 had.99. So we're we're

[4:48:14] updating those or we're computing those errors and we're going to use those to do back propagation with. Okay, so this is where kind of the math kicks in that is where kind of the math kicks in that I've talked about before is um in order

[4:48:26] I've talked about before is um in order to do the gradient descent of updating weight number five. Yeah, there was there was quite there was a lot of error. Yeah. No, definitely there was a lot of error. So, what what should make

[4:48:39] some adjustments. We're going to have to make quite a bit of adjustments because we got a pretty significant error there. Um, but what I want to zoom in on now is let's let's update this weight. Okay, let's take a look at updating this

[4:48:54] weight. And when we're going back propagation, we do have to start with propagation, we do have to start with these back uh weights. We do again we cannot start with these because mathematically we would not be able to

[4:49:07] compute those until we know what these updates are or these adjustments in terms of the gradient. So what we want to do is make an adjustment to this according to the gradient of the loss with respect to this weight. So

[4:49:24] something we need to do is compute the gradient gradient of uh the loss um with respect to um weight five. If we want to update

[4:49:37] that we need to do it with respect to weight five. So the formula for doing weight five. So the formula for doing that is here. So the the gradient which is uh the the derivative essentially which is shown by this uh quantity this

[4:49:54] which is shown by this uh quantity this delta um delta e with respect to del to delta um delta e with respect to del to to um delta 5 is um mathematically the derivative is this by what's called a chain rule of calculus. Again, I I don't

[4:50:10] think it's that important that we know all that specifically, but in order to derivative of the error with respect to output one times the derivative of output one times the derivative of output one with respect to uh the the um

[4:50:24] output one with respect to uh the the um H1 essentially the hidden layer 01 and then the the H1 with respect to weight five. So, this is all like a chain rule five. So, this is all like a chain rule kind of update. But the the thing to

[4:50:37] kind of update. But the the thing to notice here is that the the this quantity is what drives this is the quantity we use in gradient descent uh to to figure out how much of an adjustment should we make. It's totally

[4:50:53] adjustment should we make. It's totally related to this uh quantity here because related to this uh quantity here because um basically weight five is going to be adjusted um the new weight five is going to be

[4:51:05] um the new weight five is going to be adjusted by the old one minus the the um adjusted by the old one minus the the um minus the uh learning rate times this quantity the this gradient here.

[4:51:24] Okay. So we need to compute what this is. Um which is uh we're going to do that now. We're going to compute what that is. Okay. So this is all about computing the gradient uh so that we can do the back propagation. Now the in

[4:51:39] order to do that we're just going to work our way kind of backwards uh through this chain rule. So that's why it is kind of a backward propagation. We it is kind of a backward propagation. We know the total error is error 1 plus

[4:51:51] error 2. Error 1 is exactly this amount and so uh the output 01 is exactly given and so uh the output 01 is exactly given by this uh um activation. So that's that

[4:52:03] by this uh um activation. So that's that and then the this is this is computed this this is computed by this weighted sum here. So hopefully it makes sense that like taking the update with respect to five actually involves an update with

[4:52:19] respect to another weight because every weight contributes to computing the outputs of those neurons. So that's why you cannot do earlier ones without doing later ones because they all depend on each other as inputs flowing throughout

[4:52:35] from layer to layer. Right? So this is just unraveling the formula um a bit on what this error actually is so we can take its gradient. Um, again, not too concerned with us knowing all that math. Um, but mainly just wanted us to see how

[4:52:51] th those different weights enter into the picture, right? Because those we know from we just know from the way the network is structured. Um in order to

[4:53:03] compute like in order to compute anything that involves um this guy it sorry mouse is just going crazy um anything that involves this guy um is going to

[4:53:19] involve contributions from this and contributions from this. So you know in order to in order to figure out what the gradient is with respect to this um there's going to be contributions from multiple areas of the network and that

[4:53:35] what the gradient is with respect to this um there's you know contributions from uh uh contributions at this level from this weight um and ultimately

[4:53:48] contributions that I'm going to need from these and these later later on in the network to kind of um figure out what is coming out of H1 or H2. So

[4:54:00] that's why back propagation has to go this way. It has to mathematically it has to cuz as I keep saying you cannot update these without knowing what the rate of change of these are. Just mathematically you can't do it and it's

[4:54:15] because of these uh gradients. Okay. So unraveling all of this, we just take we just take uh we're just filling out this formula for our gradient descent update. Now something I should keep in mind is will we have to do this

[4:54:31] manually each time? Absolutely not. This is this is something that um TensorFlow and PyTorch will manage for us. So that's one of the beautiful things about those packages is it knows how to compute these gradient descent updates

[4:54:46] based on the graph based on the network graph. It can do all these calculations. But this is just to show you everything that's involved so we can gain an appreciation for basically everything that goes into doing uh doing a back

[4:55:02] There's a lot that's involved there which is why it usually takes time to train a neural net because these calculations are not trivial but the the

[4:55:14] network framework like PyTorch or TensorFlow manages all that for us. Okay. So plugging in everything um our error is this which makes sense. It's 1/2 of this distance plus another half of this distance. So it's just the sum

[4:55:27] of both of those. Um so in order to calculate our derivative um we're going to do piece by piece by piece we're going to compute this derivative this derivative and then this derivative. So we're just doing all of those pieces. So

[4:55:43] if you take this error and do it with respect to the output one it comes out to be this quantity. Um so we can plug in all those values and and compute this first then we can compute this then we can compute this which is going to have

[4:55:59] those contributions. So uh we have um so then we can compute this piece. So we've done this we can compute this piece next which is uh the

[4:56:13] output is this quantity um from the softmax or sorry from the sigmoid. So the derivative of it with respect to the h1 uh we can easily compute which is

[4:56:25] going to be uh this with the exponential. So it just equals this. So it's this amount. So this is all derivative calculations. Again do we need to know these derivatives? No. Tensorflow and pytorch will manage it

[4:56:38] for us. But just showing you how that's broken down. So this is this quantity. from the contributions from both of those weights um from from that part of

[4:56:52] those weights um from from that part of the network. So this uh quantity um we put everything together. So like this this uh is how we get h1. We know it's that weighted sum that goes through activation. how we uh take the rate of

[4:57:08] change of that with respect to weight five is that's actually just this coefficient um which is just this this amount. We know that we've computed that earlier. We have that value and so we can multiply all of those together and

[4:57:22] we get this as our total rate of change for the error with respect to weight five is this amount. So what that means five is this amount. So what that means is this should be used as our uh in our

[4:57:38] is this should be used as our uh in our gradient descent update of weight five. So that's going to be an important part of our gradient descent

[4:57:53] important part of our gradient descent adjustment for uh weight five. So we can compute that if we and now this is the formula I just wrote down a little bit earlier is we take the current value of weight five and we adjust it by doing

[4:58:08] the learning rate times that gradient we just computed. So the original value of the weight was 04 and we subtract off this amount. Our learning rate is a little bit high. It's 0.5. Typically it won't be that high. Usually be a

[4:58:24] fraction like 0.01. 01. But so that's a pretty significant um learning rate. But just as an example in the calculation um it just takes half of this and so our it just takes half of this and so our new weight is down here. Now that's so

[4:58:39] this is the gradient descent formula. This is a gradient descent formula for adjusting that weight. It'll be a similar formula for weight six, weight seven, weight eight, which would be um weight six would be whatever it used to

[4:58:53] weight six would be whatever it used to be minus the learning rate times the gradient of the of the loss. You see why the loss is so important. It's the gradient of the loss with respect to weight six this time. be slightly

[4:59:07] different calculations, but um we could go through all of those and figure out what the adjustments are for all of those weights. So in reality, the back propagation is going to start with all of these weights towards the back of the

[4:59:22] network. So it's going to start with all of these guys being adjusted according to that gradient descent formula. Okay, all of those guys getting Okay, all of those guys getting adjusted. And um once we do that now we

[4:59:36] can start propagating forward or I should say backwards and updating all of these guys. Should be able to do that. So we can Should be able to do that. So we can compute the um we can compute the weight

[4:59:50] adjustment for weight one weight two uh weight three weight four. We should be able to do all of those. You can see that it's going to be a similar formula. It's going to be the gradient of the error with respect to weight one. Now

[5:00:03] what's really interesting about this is again these are actually going to again these are actually going to utilize the the weight adjustments that utilize the the weight adjustments that we made in the later layers. So like the

[5:00:16] this formula is going to involve an adjustment we already computed according adjustment we already computed according to according to um output h1. we've already actually done that computation. Um so we can kind of reuse it which

[5:00:30] under the hood when we use these frameworks they are going to as much as they can efficiently reuse calculations in the back propagation they're going to reuse those adjustments that they've already been computing in order to work

[5:00:44] our way back and make all of these weight updates. We ultimately want to update every single weight. So again, we're going to start towards the end and work our way backward to adjust all these weights because those adjustments

[5:00:58] are going to reuse calculations we've already done. Okay. Okay. So um we are computing so like when we do this gradient of the error with respect to output one, it's actually the gradient of error 01 um uh times the gradient of

[5:01:15] O2 um with respect to output one. And uh we can see that there's um we're going to be reusing uh we're going to be reusing

[5:01:27] weight five. It turns out that this calculation calculation um here actually just equals weight five which we've already computed this we've already computed this as our weight five

[5:01:40] already computed this as our weight five adjustment. This is why it's so critical adjustment. This is why it's so critical to, you know, update um it's so critical to, you know, update um it's so critical to to figure out what those updates are

[5:01:52] and then use them to update earlier weights. So if you do the math on this um gradient, it actually just evens out to weight five by itself because that's this coefficient attached to this. So the derivative of it is just this. Um,

[5:02:09] and so we're just reusing stuff we've already calculated, which is great. That's what's so efficient about the back propagation. It tries to reuse stuff if it's already it's already been computed um later down in the network.

[5:02:21] So we've already we just did weight five, so we already know what that should be. Okay, so plugging everything in uh we get um the the weight five in uh we get um the the weight five value and then we uh compute just a

[5:02:35] bunch of gradients. uh continuing along like the gradient of error 2 with respect to output h1. And this gives us our total which is just this quantity uh here. So that's just one piece of the puzzle. In order to update weight one is

[5:02:50] this. In order to update weight one, we need this quantity here. Okay. So we just computed that guy. So this this part is done. And then we just need to go through and compute these two um which shouldn't be too bad. So

[5:03:05] they're going to look similar to before. Um we can do the gradient. So this this guy is particularly the sigmoid which is this. So it shouldn't be too bad to compute its gradient. Um and then this is the weighted sum which shouldn't be

[5:03:20] too bad to compute its gradient. Um and then we can make therefore our total adjustment for weight one is this. Um which is actually very very small. It's a very very small adjustment or our loss

[5:03:35] for weight one or I should say derivative of loss with respect to weight one is very very small. So it's going to be a relatively small adjustment that's going to be made to update weight one. Okay. So weight one

[5:03:49] can be updated. So uh we just go through and now the hopefully you're seeing some patterns emerge here in the sense of look at this formula. It's our usual gradient descent formula we've been using this whole time. It's the weight

[5:04:01] using this whole time. It's the weight one minus the learning rate times this loss function gradient with respect to that weight. So it turns out to be a very very small adjustment because this is a very small number. So 0.15 minus

[5:04:17] this really small number is still pretty close to 0.15. So weight one actually didn't change that much. Um weight five did. Weight 5 6 7 8 actually did change

[5:04:29] a good amount. weight one actually doesn't change too much. It didn't get adjusted by that much. And similarly, so we could go through and do all those gradient calculations and update weight two, weight three, weight four all in

[5:04:42] the same way. Okay. So we can make all those adjustments all in the same way. All right. So what we have so the takeaway from this is not the math. I about the gradient and all that because that's going to be handled by our

[5:04:57] all that math for us and we don't have to worry about that at all. What I to worry about that at all. What I really wanted us to see is this aspect this aspect of back propagation where in order to update these guys we really

[5:05:11] rely on updates that we've made previously right from the from the later on layers down in the network. We try to reuse as much of those calculations as possible and we will mainly because of how the the calculus chain rule works.

[5:05:29] It requires us to reuse those calculations in order to update these calculations in order to update these guys. So um that's why we start backwards and work our way forward. It's so much more efficient because in order

[5:05:43] to calculate this, we would need to know this calculation anyway. But we're going to do that calculation as part of the updates later down in the network. So what's great about the frameworks is they kind of store that information and

[5:05:58] reuse it in all these gradient calculations. That's one of the beautiful things about PyTorch and TensorFlow is they are so effective at these gradient calc that's what makes that's actually one of the fundamental

[5:06:12] aspects of those libraries is being able to manage back propagation. they do it really really well all these weight adjustments of these networks. Okay. Okay. So let me talk about a couple of things that can go wrong during the back

[5:06:26] propagation. So one of the things we saw is that we're computing these is that we're computing these derivatives right and what can happen is you can get these derivatives that can be very very small and close to

[5:06:43] zero. So and that can be an issue. So like when you have let me go back to this like this is a good example of a derivative a gradient of the error or the loss with respect to a weight that is pretty small. This is a very small

[5:07:00] amount. And so what that's saying is I'm going to adjust my weight by a very tiny minuscule amount. And that can make training slow down significantly because

[5:07:12] you're not making that big of an adjustment. And so the next time around, you're not you're probably not going to make that different of a prediction and you're just going to get stuck in making very small progress because your

[5:07:26] gradient's really small. Now, one of the things that contributed to this really small gradient in this example, I'll I'll tell you the main culprit. The main culprit of producing these really small values is this. The fact that we were

[5:07:42] using a sigmoid as our activation, that's the sigmoid is purposely going to squash things into fractions, which is going to make our update values pretty

[5:07:54] small when we do these because look at what we're multiplying together. We're multiplying this times this times this and we just get a tiny fraction. So and we just get a tiny fraction. So again, this um sigmoid being there in

[5:08:08] kind of our hidden layer is not this was just a dummy example, but it really causes this issue with these small gradients and um small gradients can be

[5:08:21] a a big problem because they can slow down our training quite a bit. So whenever you're using gradient descent, which every neural net training uh does, that's something we have to be on the lookout for is really tiny uh gradients.

[5:08:38] And one of the things you can, you know, like we'll be able to visualize this during our training progress is that you would you would see your loss basically stall out. So, one of the things you would see is in an equivalent graph, if

[5:08:52] you were to graph your loss over the course of your iterations, um you would see like a stalling out essentially because your loss isn't going to change that much if your gradient is really tiny, right? Because

[5:09:06] you're you're not changing your weights by that much. So, you're pretty much going to get stuck there. So, this is very um indicative of vanishing gradients. Um, so, so vanishing gradient just means really tiny descent update

[5:09:21] and it it causes you to get stuck. Now, we're going to have ways to combat this. Um, I'll talk about those later as we architect our network. One of the ways to combat this is to stay away from sigmoid and thing and like softmax in

[5:09:37] don't want to have those because you don't want to make your values smaller in the in a small range on purpose. Um, so you so that can be a huge contribution to that. So that'll be one way to combat against it. Um, we'll see

[5:09:52] just one issue that can happen during the course of training is those gradients can can become tiny which lead to very little if any weight updates and you're kind of stuck. So you want to avoid that. Now on the reverse side, you

[5:10:07] can have gradients that blow up and get out of hand. And when that happens, you basically have weight updates that are humongous. And you'll notice that too

[5:10:19] with your loss because what'll happen with the loss is maybe initially it starts decreasing, which is a good sign. That means it's learning, it's getting better, but then if your gradients are getting really huge and you're making

[5:10:31] bad updates to your weights, it can start to explode. start to explode. So exploding gradients is also a bad uh a bad thing. And usually what we see there is uh the the uh you get really

[5:10:47] large weight updates which creates a lot of instability in the training. So you'll usually see that in in your loss just becoming exponentially big because just becoming exponentially big because your weight adjustments are just so big.

[5:11:02] gradient update. You're adjusting your weight so much and you're just you start to explode in your in your error. Um so to explode in your in your error. Um so that can be bad um obviously and so uh

[5:11:15] we want a way to control that as well. Luckily we have ways to mitigate against both of these um primarily in the gradient descent algorithm that we'll use to train our network. We'll actually have a really effective way to kind of

[5:11:29] manage both of these. Uh I'll talk about that probably in lesson four when we get to it. Okay. So let's talk about gradient descent. Um so this is the gradient descent. Um so this is the algorithm that is used to minimize our

[5:11:42] algorithm that is used to minimize our loss function particularly to to tell us how much we should adjust our weights in order to minimize that loss function and and it works by deriving this kind of formula right which says I should be

[5:11:56] taking a step in the direction of the gradient um of the loss function in order to minimize that loss function. Um, so we've kind of seen that and the picture that I think makes the most sense for descent is thinking about our

[5:12:13] loss function as kind of a parabola like this. And we're trying to work our way towards the minimum of it. And we do that by taking steps in the direction of the slope, right? The descent of the slope. So we're taking steps this way.

[5:12:28] Um, that slope is represented by the gradient and we're taking our steps down towards the minimum. hopefully reaching that minimum. Now, one of the issues that can happen with gradient descent is that it the loss function is not a

[5:12:43] perfect parabola. We would love it to be that way. Um but it's not always that that way. Um but it's not always that way. um it can be jagged and kind of have multiple um multiple peaks and valleys which can create a challenge

[5:12:57] can kind of get stuck thinking that you've reached a minimum but you haven't reached the true minimum of the loss. So your model is not as good as it could be. That's something that can happen with um gradient descent especially if

[5:13:12] you have a complex loss function. You can kind of get stuck. Um but we're going to we'll talk about how to kind of mitigate against that as well. Okay, so let's talk about gradient descent um in the context of linear regression. So you

[5:13:24] guys remember linear regression. It's just finding a simple model like this. Maybe it only has one feature. So it only has one weight and a bias. This would be most basic linear regression there is. Um and we're trying to find

[5:13:36] the best fit model, right? The best we're trying to find out what these parameters should be, the the M and the C. uh in order to get that best fit line through our data points, right? That was linear regression. Um so linear

[5:13:52] linear regression. Um so linear regression, you know, is um used to train or sorry, gradient descent is used to train linear regression actually behind the scenes because there you can iteratively go through and find out what

[5:14:05] those weights should be through a gradient descent update on those weights in the exact same way we just did gradient descent for the neural net. It's just not as complicated because there's only a couple of parameters, the

[5:14:19] the slope and the intercept in this case. And um I don't need to do back propagation to update them because it's just a simple derivative. Um so it's actually really easy to do that. Um, so if our loss function is MSE, the mean

[5:14:36] squared error, it's actually really easy to find the the minimum of that loss function by taking steps down this parabola. So exactly the picture I had before, we are going to descend and take steps

[5:14:50] towards the minimum until we reach the minimum and find the best parameters that minimize this loss function. Okay, the goal of the training and we know that's what it's doing, right? It's

[5:15:05] fitting it's trying to find the best betas or the C and the M. Uh, and that's we're doing with neural nets. We're trying to find the best weights that

[5:15:17] minimize this loss function. And we're going to take steps towards the bottom by taking the gradient of this loss function. Uh, because the gradient is is the slope. We want to move in the direction of that decreasing slope here

[5:15:32] to get towards the bottom. We want to take steps towards that. So this so algorithm. It's an optimization algorithm that iteratively updates weights to minimize the loss. That's the goal. Minimize the loss. So we should

[5:15:46] see it decreasing over the course of the training. Um and we want to move in that direction of that slope, that gradient. Um so we update the weight. We know there's a formula there that updates the weight by taking a step proportional to

[5:16:01] the gradient, which is which is really nice. That allows us to kind of make our way in steps towards the minimum of the loss. And along the way of doing that, we're making these weight adjustments so that our model is getting better and

[5:16:15] better and better and producing less loss each time. That's the goal. So like I said before, the issue that can happen is the loss can be a complex function um

[5:16:29] and not as straightforward as like a mean squared error. Um and so it can be difficult to find our way towards the best optimum minimum, which is a challenge with gradient descent in general is you're not necessarily

[5:16:43] general is you're not necessarily guaranteed to find the lowest minimum. So you may find what looks like a good minimum um but it may not always be the optimal minimum which can be a risk. But again we're going to have ways to kind

[5:16:57] of combat that and try to find the optimal minimum as best as we can. So now think about it from the opposite perspective of finding a maximum. So if we were looking for a maximum, we would actually do what's called gradient

[5:17:12] ascent. meaning we would work our way up. Like if we were trying to reach the top of the mountain, we would take steps that slope us upward, right? We would want to go upward. So it's kind of the same like if you want to maximize a

[5:17:26] function, you actually do gradient ascent, which should go upwards towards a peak. That's what you would want to f find, right? Is a peak. If you're trying to do gradient descent, you're trying to go down because you're minimizing

[5:17:40] something, which is usually what we're doing with a loss function is minimizing that loss. But this just to call out like gradient descent actually could be used to maximize something by taking a step towards the and kind of towards the

[5:17:55] step towards the and kind of towards the the slope um going upward. We could do that. Okay. So you can see we're kind of now the question is like when do we now the question is like when do we converge essentially um the convergence

[5:18:07] is once we've done enough steps. So uh now we'll be able to actually have other um ending or stopping criteria which will say like okay yes we want to we

[5:18:19] want to converge if we reach a certain number of steps in our descent but we could also converge if we start to see that our loss is is diverging away from

[5:18:33] uh where we want it to be. So, like if we're if our loss is decreasing at a good rate and then it starts to turn upward, maybe we want to say, "Okay, let's cut it off here because it looks like it's getting worse." Um, so that's

[5:18:46] something we'll do later on is see how we can set that up as kind of like a convergence criteria. So maybe we want to keep going, keep going, keep going, little worse. That's where we can kind of cut it off and stop. Or we could just

[5:19:00] reach a maximum number of steps and stop then. So, but we'll we'll be able to evaluate that uh visually, which will be really nice during the training process um using TensorFlow or PyTorch, they they have uh built-in ways of kind of

[5:19:15] evaluating your progress during the training and controlling that uh training and controlling that uh optimization. Okay. So, again, in the uh in the neural network land, what we're trying to do is change our weights. Uh

[5:19:29] and usually we're using gradient descent because we want to find the minimum of a loss function. So we're we're evaluating our loss taking a step that is going to decrease it, right? Taking a step in the direction of that slope that decreases

[5:19:44] the loss function. So it's this picture. I've been using this picture quite a bit, but uh here it is again that the learning the learning rate enters the picture in the gradient descent to kind of control how much of a step should we

[5:19:58] take in the direction of the gradient. So the gradient gives us generally we should be going downwards in this direction of the gradient. But the um learning rate says well we should take a proportion of that gradient value and go

[5:20:13] you know this amount down and then take another step this amount down and keep doing it that way so that we don't risk overshooting the minimum and jumping all the way over here and then if we take too big of a step we could jump back and

[5:20:28] forth and just keep jumping back and forth over and over and over again and we'll never actually reach the minimum. So uh the learning rate is a really important parameter in gradient descent. Um it's typically chosen to be something

[5:20:41] Um it's typically chosen to be something small like 0.01 is a very uh industry standard value or even something smaller like uh 0.01. Another thing to realize about learning rate is it doesn't have to be constant during the whole

[5:20:56] training. It can actually be changed. So there are some strategies and we'll learn about this later on. There are some strategies that um basically give us a schedule over the course of the iterations to say okay maybe the

[5:21:09] learning rate starts off high in the initial first handful of iterations and then later on it actually slows down and the learning rate um becomes smaller and

[5:21:21] smaller and smaller. And the idea behind that is maybe initially like when your error is bound to be the highest because you have random weights um you're willing to take a larger step. Right? So initially you're willing to take a

[5:21:35] and you're willing to take big steps initially but then as you do enough steps you start to slow down and take much smaller steps. So there are things like that that we'll study um that it it can be variable. It

[5:21:52] can be like uh um you can do like a decay an exponential decay essentially steps I'm going to start decaying this so I encourage smaller steps as I've as I've taken more and more and more. So we'll study that later on. All right. So

[5:22:07] just to wrap up well first of all were there any questions on gradient descent. So, so just to recap that gradient descent is our algorithm for training a neural net. It's what it's what is going to optimize our weights. The thing that

[5:22:21] does the weight adjustment is the back propagation. The thing that tells us how much we should adjust is gradient descent, right? It's what's telling us, it's giving us that formula so we know how much we should adjust that uh adjust

[5:22:35] those weights. Okay, so just to recap on the perceptron before we [snorts] move on to lesson four. The perceptron is a really simple algorithm, right? It's it has a single neuron that takes a weighted sum through an activation and

[5:22:50] produces a zero or a one based on that activation. Um, so it's limited. It can only produce a zero or one. It cannot produce a probability, which is bad for a lot of situations, right? It can only produce a zero or one. So it's not going

[5:23:05] to be useful in multiclass classification for sure. Um but uh it it is kind of the most basic neural net that there is in a sense of it's modeling a single neuron taking a weighted sum through an activation. The

[5:23:19] other thing is we saw this before the perceptron only works with linear data. perceptron only works with linear data. So as as a model the perceptron is very limited in what it can actually model. Um so whenever the classes are kind of

[5:23:34] arranged in a nonlinear separable way um you tend to produce a lot of error. Um so you really what this is signaling is you really need a more complex network of neurons right you need that network so you can learn nonlinear

[5:23:50] relationships. That's what you really need uh beyond just a basic perceptron. So like not every situation is going to line up perfectly like this which is you know where okay everything can be separated by this uh line and so

[5:24:06] everything on this side's the other class um and essentially what this line is is this dotproduct and seeing if you're bigger than zero you're on this side if you're less than zero you're on

[5:24:20] the perceptron's doing in terms of its activation. So, but not everything is like this, right? In fact, most data is not. And so, that's where you need a neural network with more layers and more neurons to pick up on those nonlinear

[5:24:36] neurons to pick up on those nonlinear patterns. So basic perceptron very limited but it what I'm hoping we got out of this lesson is it serves as the can take weighted sums through activations and stitch together many

[5:24:51] many neurons and learn nonlinear relationships very effectively. So relationships very effectively. So again, the perceptron fails to um be optimized, fails to converge. Usually basically you end up with an endless

[5:25:05] cycle of weight updates because you're never going to converge uh unless you your model is not going to be very great whenever the data is not linearly separable, which is most of the time. Okay. So for this lesson um what we

[5:25:22] basic building block for neural nets because it models a neuron. It takes it it through an activation. We are going to build off of that with larger and

[5:25:34] larger and larger networks. What we're seeing is we really need that power of those big networks in order to learn nonlinear relationships. So perceptron's just not going to cut it for most problems because it's so limited. Um but

[5:25:49] it's really good inspiration uh because we want that style of computation to persist in a larger network but just with more activations, right? More nonlinear relationships. Okay. So what are we hoping to get out

[5:26:05] of lesson four? Well, we want to just expand our knowledge about perceptrons to larger and larger networks. So, we're going to talk more about the um things that we need to do to get there, including uh setting up our loss

[5:26:19] functions. Um so, see what kind of different loss functions that we'll use. We've kind of already studied like mean squared error we know is there, but just want to review like for classification, what kind of loss functions would we

[5:26:31] use, which are normally going to be along the lines of those cross entropy ones that we've used uh in logistic regression. So, very similar. And then we'll talk about the forward and backward propagation that occurs in deep

[5:26:45] neural networks and see kind of what the difference is to uh what we've been talking about. Really no difference except now the network is bigger and so there's more more neurons at play, more activations, more gradients that have to

[5:27:00] be computed, those kind of things. So just expanding everything we've done to uh deeper networks here. And this the point of this lesson two is to prepare us for putting all this together and building our own neural nets in which

[5:27:15] will be in lesson five and six. So lesson five and six will be all about uh TensorFlow and PyTorch. So probably on Wednesday uh we'll finish up this lesson and then move on into lesson five where we'll build our own neural network

[5:27:30] inside of TensorFlow. We'll get a good feel for that um and and do a lot of examples there. So Wednesday will be pretty good for building our own. Okay. pretty good for building our own. Okay. So talking about the uh deep networks

[5:27:43] So talking about the uh deep networks here. Um now this is just a referral to the how big the network is in terms of its hidden layers. So if it's got a lot of hidden layers um that means it's a deeper network than than having very few

[5:27:59] deeper network than than having very few which will kind of be a shallow network. which will kind of be a shallow network. Um, so the the deep neural networks um are going to have a lot more neurons. They're going to have a lot more layers.

[5:28:11] weights. And one of the consequences of that is they're going to be more intensive to train because there's going to be a lot more weights to learn the more hidden layers that we have. Um, where things are connected together in

[5:28:26] more weights to to deal with and a lot more weights to update. Okay. So on the flip side of that though, the deeper the network, the more powerful it can potentially learn patterns. So um you

[5:28:40] know we we want things to be reasonably deep for the problem at hand. So like when we get into computer vision for example um those are going to require somewhat sophisticated deep networks because learning how to do things like

[5:28:55] object detection or image recognition are difficult problems. Um, same thing with text and things like text generation and generative AI um, in transformers. Those have to be deep networks because they're representing

[5:29:10] really complex uh, relationships. Um, kind of like what we see with language understanding in a GPT, right? It's they're very complex networks. So, they're very deep networks have lots of layers, lots of weights. As I said,

[5:29:24] modern transformers um are pushing into the hundreds of billions of weights. So, it's a really massive network uh really deep network.

[5:29:36] So, going back to computer vision, you know, um we may have we may be looking at images of different dog breeds and we may want to predict which dog it is. So that we were choosing from there may be you know three probabilities that are

[5:29:52] produced and that's an example where you know we may have a really sophisticated deep network but in the final layer of it um there's going to be three neurons and it's likely going to be a soft max output uh in terms of the activation

[5:30:09] because we want to produce three probabilities for three different breeds and we want to produce probabilities um uh generally that add up to one. So this and this this one here should be something like 0.2

[5:30:24] um sorry no.3 which is which is what it is. Sorry I read that as 2. This is a.3. is. Sorry I read that as 2. This is a.3. So this this should total up to one um which it does. So this should total up to one and they should be

[5:30:39] probabilities between 01 if we were doing that kind of example. So again, it could be a really sophisticated network, but the final layer is something like a softmax activation that's producing these probabilities for our classes. Um,

[5:30:53] and we'll see that when we get into computer vision. Um, we may have audio. So we may have a deep network that's um analyzing sound data. So like uh uh signals like waves and those could be

[5:31:08] categorized into different uh like music could be put into different genres. It could be put into different sound types. Um and again this would probably be a very sophisticated deep network and it would be very difficult to pull

[5:31:21] something like this off without a deep network. So we've talked about this before. Neural nets give us this they unlock the capability to work with this you know advanced data like uh audio images text video things like that. Um

[5:31:37] so deep networks are present in in so many areas like self-driving cars image voice recognition chat bots voice assistance online translation. I just gave an example you know of the chat bots uh where we're going to learn about

[5:31:52] that later on the transformer network. Those are very deep networks that have enabled, you know, generative AI in the most recent years a and uh really

[5:32:04] very complex. They they're very complex. the other thing is with some of these deep networks is they involve different types of computation than what we've studied so far. So what we see in

[5:32:18] like computer vision is convolutional layers which are advanced types of computations that are beyond just doing a weighted sum through an activation. basic type of computation a neural network can do just a weighted sum

[5:32:34] through an activation which is what we've studied to this point. Right? But we there's going to be a lot more advanced types of calculations like in a advanced types of calculations like in a in a transformer there's attention

[5:32:46] attention layers which are doing you know an advanced calculation to determine how words should be related to each other so we can understand context each other so we can understand context and meaning. Um so a lot to come but uh

[5:32:59] these are all powered by kind of deep neural nets. All right. So, when we neural nets. All right. So, when we start to put together our own deep nets, um, one of the things we're going to need to do is define our loss function

[5:33:12] so that we can train it, right? So, we can, of course, we'll build the architecture of them. We'll learn how to do that in TensorFlow and PyTorch. Um, but in order to train it, we will need to specify a loss function. So, we'll

[5:33:27] need to understand how to do that. And so we're going to go through um some examples of loss functions here. Now, we just we just wrapped up talking about loss functions. So I don't really need to spend a lot of time on what it is. We

[5:33:40] kind of know that it's measuring the error. What we want to do is find the minimum of that through gradient descent, right? Uh and minimize that descent, right? Uh and minimize that error um during the training process. So

[5:33:54] we've talked about that and what we're going to do is talk about different types of loss functions that we typically see in uh training of deep networks. Now one example we've used before is you know you could one very

[5:34:10] simple loss function is just take the difference between um you know take the difference between like the probability let's say and what the label is. So in this in this example um that's just a simple difference but this is not a very

[5:34:26] good loss function for training. So we're going to learn like what a what a better loss function would be um for doing classification. So just subtracting like zero or one not the most effective. Um we're going to use

[5:34:40] something more advanced than that that lends itself to a better like gradient descent update than just taking a difference. Uh so we'll see that in uh coming up shortly. But generally we know the loss function is capturing the error

[5:34:54] uh between our model our network's output and what the true label is right that's what the loss function is doing. So generally you know the loss function can be broken down into uh the loss that we use for a regression problem versus

[5:35:08] And we've actually been we've already looked at the regression loss for the most part. Uh but we what we're going to focus our attention on um is mostly on classification laws because a lot of uh a lot of deep learning problems actually

[5:35:25] are more classification than they are regression. Um like object detection or predicting what word should be next when we're generating text from an LLM or something like that. um most of it is class mean squared error as a really

[5:35:41] effective regression loss function. Um so we we know all about that. We've been studying that before. So nothing really new to say there. Um again it's measuring most of the time it's measuring that distance, right? It's

[5:35:55] measuring the distance from the label to our predicted point. That's what these regression loss functions tend to do. And the ones that are most commonly used for regression um are mean squared error and mean

[5:36:09] absolute error. Now we've looked at both of these before and again they show up here in deep learning because we need some type of loss function to measure our error use that to make a gradient descent adjustment to our weights. Um

[5:36:24] but these are the two options that are primarily used in uh deep learning as well. We've seen these before with things like linear regression. Not much of a difference here. We can either use mean

[5:36:37] squared error or mean absolute error. So the mean absolute error as a function kind of looks like this where it's taking the average of these distances. Now the distances are computed a little bit different than um taking the square

[5:36:51] distance. They're using just the absolute value difference. So that's that's a little bit um different than doing the squared which is more of a natural kind of distance calculation there. So uh this that's one of the

[5:37:06] that's the primary difference really is doing the absolute value of those rather than the square. So we get that for our loss function kind of that average um

[5:37:18] distance in terms of absolute value. And so if we were doing an example, we could compute the uh average error by just taking the difference between the uh what our network says the value should be um which would be like these

[5:37:33] predicted values and what the actual value is for each of these examples. difference and then we average those together and we get like a mean absolute error of 088 which is pretty good. That's pretty low. But you know that

[5:37:48] that'll be up to us if we were doing a regression which loss function we would want to use. We could use this. We could use mean squared error. In code, it would kind of look like this. Now when when we um when we use this in

[5:38:02] TensorFlow or in PyTorch, it's going to be much different. We'll be able to specify that loss function as part of our training code uh when we get to that point. So, but this is just to show you like if we were to do this in numpy, um

[5:38:17] it would just be taking the absolute value and then taking the average of those uh those absolute errors. Now, the mean squared error, and this is a typo, this should be parentheses here.

[5:38:30] Not really a big deal because it's, you know, going to pretty much be the same, Now, this is the mean squared error we've been using all along. Um, so we take the squared distance between our our actual target and our predicted

[5:38:45] value from our network and this gives us our loss function which we average this across all of our data. Um, and so that's a this is a very standard loss function for regression. So this one is more widely used. Probably 99% of the

[5:39:01] time mean squared error is going to be used for for a regression problem. Uh, absolute error is rarely used. Um, the only time you would use it is if you're very certain that you have a lot of outliers for some reason. Like you have

[5:39:16] a lot of outliers and you don't want those outliers to be amplified as much. And the mean squared error, you could use mean absolute error. But for the most part, I think the mean squared error is a very common loss function for

[5:39:28] regression. So obviously what we do is we subtract, we find the distances. we've talked about this over and over again is like we take the distances here and we square them uh and then we just average those. So for that same data,

[5:39:45] our mean squared error is actually a little bit higher. Uh and that's be and that makes sense. That's because the uh errors are kind of amplified when we square them rather than just taking the single absolute value. So here like if

[5:40:01] we were to do this manually in code just in numpy it would kind of look like this where we take the uh actual minus predicted we square that we just uh in python remember that's star star 2 is squaring it um and then we take the

[5:40:17] average of those squared errors so that makes sense again like when we define our loss function in in our tensorflow or pietorch it'll look much different than this this is just to kind of explain the calculation

[5:40:30] Okay, so you can see uh there the errors are a little bit more amplified. Now, from a training perspective of the neural net, that's not actually a bad preferred because that's going to encourage us to make better adjustments.

[5:40:45] Um, you know, potentially better adjustments on our weights. Uh, we have a little bit more error to go off of in the squared case. So, so that's actually the squared case. So, so that's actually fine in that situation. But you can see

[5:40:58] like if you have an outlier um the square is really going to amplify it. So this is an extreme case where we change the predicted value and so we change the actual value to something much bigger. So this 31.5 is likely an outlier

[5:41:14] compared to the rest of the data. So when you have an outlier label like that um your prediction can be significantly amplified in terms of its error. So this this produces just a humongous MSE whereas it doesn't affect

[5:41:30] humongous MSE whereas it doesn't affect the MAE quite as much. So again like if you know if you if you suspect there's a lot of outliers in in the data that's where the MAE may be you know better as a loss function wouldn't amplify those

[5:41:44] as much. Okay. So just kind of showing like for data without if we have data that doesn't have outliers like on the right MSE is going to be our preferred problem. We're going to lean towards MSE.

[5:41:59] It's a very one of the reasons by the way one of the reasons mean squared error is preferred. If we go back to the formula for it um this square is very desirable from a gradient perspective. If you think about like because a square

[5:42:14] is like a parabola, right? So having this square is actually really powerful from a loss perspective. So our loss much easier to take steps towards the minimum and and basically guarantee we

[5:42:28] only have one minimum there. So it's desirable to have that square. Okay. So again, this is why mean squared error, if you're doing if you're trying to squared error is going to be your preferred loss function. One of the

[5:42:44] reasons is it's so effective in back propagation for that reason that the square is really nice. It the square makes the gradients really easy. The absolute value actually is a difficult gradient to do um because it it

[5:42:58] technically doesn't have a gradient in certain places um mathematically. So um this is a much easier function to work with under the hood for computing like

[5:43:10] those gradient updates. So the squared is actually preferred in in the loss is actually preferred in in the loss function. Okay. So that leads us into uh classification. So we talked about regression. If we're doing regression

[5:43:25] and we're training a neural net, we want to use mean squared error as our loss function. That'll make uh training it really easy uh or not I shouldn't say really easy but much better than if we did mean absolute error. Let's talk

[5:43:38] did mean absolute error. Let's talk about classification and kind of um talk loss function there. So let's start with the most basic case of binary and then we can work our way up to multiclass. So with a binary classification

[5:43:53] um we are checking uh we're we're trying to compute our error where the the classification is going to be into only one of two options right it's going to one of two options right it's going to be a cancer not cancer fraud not fraud

[5:44:07] it's going to be a one or a zero and it's only one of those two options right so in a binary classification those are the only two options we have but in multiclass we know it's more complex than that. So in a multiclass there

[5:44:21] could be more than two. Um we could be doing apples, bananas, grapes or the example I've always used is with images like cats, dogs, giraffes. Um but it could be many many categories that we have. And what we do in multiclass

[5:44:36] classification is usually predict the probability of belonging to each class. Right? So that's where that softmax comes in in the end of the network to kind of see see what is the probability of each class. So the reason to mention

[5:44:52] this is that's going to dictate what kind of loss function we use for classification. If we're doing binary or if we're doing multiclass, um there's going to be a slightly different loss function uh for each of

[5:45:05] these. Honestly, it the multiclass is just really a multi-dimensional extension of the binary case. So binary case is really easy and the multiclass is just an extension of that in terms of the loss function. Okay, so for

[5:45:21] classification, we have two kind of main categories of loss functions. And these important because these are what we're when we do training of neural networks, especially for classification. Um we are going to need to define this um in order

[5:45:39] to train it like what loss function we want to use to compute our error. And then that dictates what kind of adjustments we should be making to our weight. Right? So the two options that

[5:45:52] we have are going to be cross entropy and hinge loss. We're going to actually focus our attention on cross entropy because this is the one that's most widely used in classification. Um hinge loss is rarely used. I I haven't really

[5:46:08] seen it used too much, but we'll talk about what it is and how it's a little bit different. But uh we'll focus our attention mainly on cross entropy widely used for classification. All right. So um won't be able to finish

[5:46:22] this today because we only have about a minute left. But where we're headed with minute left. But where we're headed with this is we need some way of keeping track of our error when we're producing

[5:46:34] probabilities. This is this is where um we need a special function to kind of penalize us putting probability on the wrong result. So we know like just to

[5:46:46] wrap this up, we know that for a classification, especially multiclass, it's only going to be one of these three options as the actual label. So um it's either going to be Samsung, Apple, or LG as the category of phone. Let's say we

[5:47:02] were analyzing a picture of a phone and putting into one of these three uh uh products, Samsung, Apple, or LG. It's only going to be one of those three. But our model is going to produce a probability for each one of those. And

[5:47:16] so what we need to do is have some type of function that can compute the error if we know the actual category and we know the probability. So, and what we

[5:47:28] want to do is penalize situations where we're putting our probability, like a high probability on the wrong class. So, let's say we put a really high let's say we put a really high probability on LG, like a 0.9, and it

[5:47:40] actually was an Apple. So, it was a 1 Z as the class, right, for Apple. And we had a 0.9, we had a really low had a 0.9, we had a really low probability for Apple, like maybe we had

[5:47:53] probability for Apple, like maybe we had uh 07 and then we had 0.03. And so we this and we need some type of function to do that. Okay. So to get an intuition for cross entropy loss. Now this is again going to um be a loss function to

[5:48:09] going to capture this error on classification is to basically look at classification is to basically look at the difference in probability. our model is going to be producing a probability um more often than not especially in the

[5:48:24] very typical multiclass classification where we're predicting amongst where we're predicting amongst potentially um many categories right so potentially um many categories right so in that situation our model produces

[5:48:37] something like this on the in the orange on the right it produces something like this which is uh basically a distribution right to say okay I've produced do some probability of this class, let's say it's Samsung, some

[5:48:51] it's Apple, and then some probability of this class, which is LG. And those need to add up to one. And of course, these are all fractions between zero and one. And we generate this from our network, right? And and usually this gets

[5:49:06] generated from a softmax activation that's in the kind of final layer, right? And so this is a pretty typical situation. Maybe it's three categories. we produce three probabilities. That's pretty typical. So

[5:49:19] probabilities. That's pretty typical. So in order to measure our error, we want to compare this to the true distribution. The true distribution is going to be where there's a 100% probability on the actual class. So if

[5:49:32] probability on the actual class. So if it's like a Samsung phone, then I what I should get for the true. So for the for the label, let's say, or the ground truth, whatever you want to call that, the label is going to be a distribution

[5:49:45] as well, but it's it's going to have ones and zeros, right? So like if it's actually a Samsung, there should be a 100% probability there and then zero for the other guys. That would be a pretty typical like label distribution, right?

[5:50:00] Would be something like that. And what the cross entropy is going to do is measure kind of the difference between these two distributions almost like a distance. It's going to it's going to measure the difference between these two

[5:50:15] and generate a penalty whenever we have our probabilities that are different than this true label. Because what would be a perfect prediction is if our our be a perfect prediction is if our our output was 100% Samsung, 0% Apple, 0%

[5:50:29] LG. That would be a perfect prediction. is something that's very close to that might be something like where where we have 90% of our probability on Samsung have 90% of our probability on Samsung and then like 5% on Apple and another 5%

[5:50:43] on LG. That might be another that might be a prediction which is pretty good. I mean that's saying we're pretty confident this is a Samsung and in in generate a prediction we take the maximum probability. So 0.9 is by far

[5:50:59] the maximum right? So we would we would predict Samsung for this. But this gives us a way to kind of measure our error the cross entropy. It's going to measure mathematically the difference between this prediction in terms of

[5:51:15] probabilities and this true label distribution. That's the goal of the cross entropy. So it's it's kind of like mean squared error um in terms of measuring a distance. It's just measuring like probability distances.

[5:51:30] So, think of it that way. Okay. So, we're going to I'm going to show you the formula coming up, but that's the intuition is to is the fact that we need a way to measure how far off our probabilities are in our prediction,

[5:51:44] right? That's that's kind of the goal of the cross entropy. Okay. So, with that the cross entropy. Okay. So, with that in mind, let's go forward. So again the actual um the actual labels would look something like these guys right on the

[5:51:57] right. So these would be pretty typical for our data where we have our labels um something like this depending on which class that that uh example is. So if it's like an image and it's a Samsung phone in the image um it should be a

[5:52:14] 100% probability like we just said. If it's an apple, it should be 100% on the second uh entry. If it's an LG, it'd be 100% on the third. And we're going to produce these probabilities in this uh output and kind of measure that distance

[5:52:31] from this to these kind of labels. That's going to be the goal is kind of measure that distance there. Okay. So, here's an example again where we uh now

[5:52:43] in this example, we're not quite as confident. So let's say it's actually an LG. So therefore there's 100% probability in this third slot. Zero and zero here, which makes sense. This is this is the label, right? This is the

[5:52:57] label here. Let's say in this example, that was the label. But our model that was the label. But our model predicted 60% LG, 30% Apple, 10% Samsung. Right? These were the probabilities that came out of our our

[5:53:10] probabilities that came out of our our prediction. Now from a from a prediction perspective, we would get it right because technically if we take the maximum of this, we would still predict LG, but what we're really after

[5:53:25] is how good is our model because that's going to dictate adjusting our weights because what we would love is to get this very confidently correct. We barely get it correct because this 6 is just enough to be more than the others. So

[5:53:40] yes, we would predict LG in in a and realistically, but it's we want a true kind of mathematical distance there between these two uh like our prediction and our label just like what we do with regression, but we're doing it with our

[5:53:55] probabilities this time, right? Um hopefully that makes sense when you see it this way. And so the intuition should be that there should be a decent penalty for this because yes, we're getting it right, but not confidently correct. So

[5:54:12] there should be some level of penalty for this versus if we had way more confidence in LG. Like let's say we had even more confidence like.99.

[5:54:24] So we had that and then maybe we had 1% on Apple and then 0% on Samsung. Let's say we had that. The intuition should be that this setup should be penalized much

[5:54:36] less because we are very very confident it's LG. The model's doing a pretty good job. It's 99% confident that it it's that the prediction should be LG. So it's it's put all of its probability nearly all of the probability on LG.

[5:54:51] That's a pretty good prediction. this would be desirable and this should be penalized less than a situation like this with the 6.3.1 because it's not as probability that's assigned to the other labels and it's only 60% on LG. This

[5:55:08] labels and it's only 60% on LG. This should be a higher penalty than than 0.99.01 and zero which is a pretty good prediction. It's nearly perfect, right? Nearly perfect. So to mathematically

[5:55:23] capture this we compute the cross entropy. So in terms of a formula um it looks like this. It's actually a very simple formula which is to multiply the

[5:55:35] simple formula which is to multiply the uh to multiply the labels times the logarithm of the probabilities uh uh for those labels. So if we have uh let's say

[5:55:47] [clears throat] we had um by the way this formula is here um and you might be wondering why is there a negative in front of all this? The the reason these uh values here are all of our

[5:56:03] uh values here are all of our probabilities, right? And so um they're going to be fractions that are less than one. Anytime you take a logarithm of negative number. So these are all going to be negatives. These are all going to

[5:56:16] be negatives because they're logarithms of fractions. And so you're going to you're going to basically take the negative of a bunch of negatives, which are which is going to make a positive loss, which is which makes sense. There

[5:56:28] should be a positive error. Um kind of like a distance, right? There should be like a distance, right? There should be a positive distance uh there. And what a positive distance uh there. And what you can also see from this calculation

[5:56:41] is this calculation actually simplifies quite a bit. If you think about it, um quite a bit. If you think about it, um what's going to happen in the uh in the label is only one of these guys is going to be one. We saw that with our LG,

[5:56:56] Apple, Samsung example, right? Only one of these guys is going to be a one. The others are going to be zero. So what you're going to see in this formula is it actually reduces down quite a bit because the the things that are zero are

[5:57:11] essentially going to be canceled out. Right? So like if if the true label was a one for the first class and then zero and then zero and then zero for all the other classes. Um that's essentially going to wipe out all of these terms,

[5:57:25] right? And so you're really only going to get a contribution um from the you're only going to get a contribution from this term from this probability which is why you would really like that probability to be high.

[5:57:39] The closer that probability is to one the logarithm of one is the logarithm of exactly one is zero. So if if if this were one as well and we had 100%

[5:57:51] probability on this then the error would actually be zero because it would just be 1 time zero. But something that's really close to one like a.99 if you take the logarithm of.99 um that's going to be pretty close to

[5:58:05] zero a very small number. So it's going to be a small penalty to be a small penalty if you get it way off. So let's say um let's say it's the class is one but you predicted a very low probability for

[5:58:18] that. That'd be a mistake. That'd be a pretty bad mistake, right? So let's say you only predicted a 0.1 probability for that and you had all your other probabilities on these other classes, right? And the true label was a one and

[5:58:31] then 0 0 0 etc. The issue with that is yes all of these guys are zero but if you look at this term this is a one but the logarithm of

[5:58:43] this really small fraction like point if you have a low probability there is going to be a big error the logarithm of that's going to be a pretty big negative number the close the smaller it is it's a larger negative number and so um and

[5:58:59] a larger negative number and so um and so then you get a much bigger your so then you get a much bigger your contribution to your error to your loss. Okay, so mathematically that's why you have logarithm there is because you want

[5:59:12] have logarithm there is because you want whenever you have a small probability whenever you have a small probability you want that contribution to be more um for the target, right? You you would like this to line up so that most of

[5:59:25] target and that's a very small contribution to the cross entropy. Okay, great. So that's the formula for that. Now will we need to calculate this by ourselves? Of course not. This will be handled by TensorFlow or PyTorch. But

[5:59:38] this is just the intuition for you know why we use this formula under the hood to do our training. Um remember this is our loss function. So this is the function that's going to be used during the gradient descent training of our

[5:59:52] neural network. Um and this is particularly useful for classification. Right? we had mean squared error which is very natural for regression. Um but in this setting of classification we need to use this uh loss function cross

[6:00:07] entropy. Okay. So here's an example calculation. If we go back to um that setup we had before where the actual label was LG and we had these

[6:00:19] label was LG and we had these probabilities where we had um 6.3 probabilities where we had um 6.3 and 0.1. Um if we were to calculate this cross entropy, it looks a lot like this. So we have um again like I said really

[6:00:33] only the contribution from this term that where the label is on the LG. And so these other terms don't contribute anything because they have a zero there. anything because they have a zero there. But what we can see is the logarithm of

[6:00:47] 6 is going to be a negative number. And therefore when we take the negative of therefore when we take the negative of this negative number we get a positive error contribution which makes sense there there is some level of error there

[6:01:00] and that error is going to be more than if we had.99 sitting here because the logarithm of.99 is going to be a number that's much closer um to zero. Okay, it in case you're in

[6:01:16] case you're wondering, the logarithm um the logarithm graph kind of looks um the logarithm graph kind of looks like this. Um and it crosses this axis like this. Um and it crosses this axis right here at one. Um so.99

[6:01:37] uh 0.99 is going to be a very tiny value that's very close to zero. um whereas 6 is going to be somewhere down here um that's going to contribute more error. Okay. So um given that we've talked about cross entropy the the cross

[6:01:52] two different ways. The one that we just studied is the categorical cross entropy studied is the categorical cross entropy loss um which is used for multiclass classification. There's a simpler version of it whenever you have a binary

[6:02:07] meaning you only have one probability which is for between two classes like a spam not spam or cancer not cancer. It actually reduces even further into a simpler formula. But the one we just studied is is very

[6:02:22] much related to categorical cross entropy which is uh mainly for the multiclass. So the reason it's called categorical is because it's for multiple categories. Uh so so this is the one that would be used most of the time for

[6:02:36] that would be used most of the time for multiclass classification and this the binary cross entropy is going to be a reduced simpler version that's going to be uh only in the case of binary uh classification.

[6:02:50] So we only have two classes and we produce a single probability uh and the label is only going to be a zero or a one, right? There's only two options. Um, so the one we just did is where there's, you know, we have a prob we

[6:03:03] have a a multi-class classification. So there's multiple probabilities at play. there's multiple probabilities at play. So that's going to be this one.

[6:03:15] Okay. So let's study those. So if we were to look at some various cases, um, I want you to, this is kind of getting at what we were just talking about um, earlier. So if we're looking at all of our data, so let's say we went

[6:03:29] through every single example, every picture that we had in order to compute the loss for that iteration, we're going to we're going to basically average those errors together. So that's what's actually going to happen with the

[6:03:42] categorical cross entropy is we're basically going to average all those cross entropies together. So if you look at this table, you can see some scenarios where there's much higher error in some cases. I want I

[6:03:57] want to pinpoint a couple of cases that kind of jump out to us. Um based on what we were just talking about, look at this case here where we produce a 2.3 error. That's a significant error. Why did we produce that? If you look at this uh

[6:04:12] produce that? If you look at this uh example, the true label was apple. So there's a one here in the second slot. That's an Apple prediction, right? An Apple label. There's a one there in that second slot. But what did we predict? We

[6:04:25] would have gotten this wrong and we would have predicted Samsung, right? Because the most of the probability here for this that our model might have produced would would have been here on the 6. Um we only had a probability of

[6:04:41] mistake, right? That's a very low probability probability um for the actual result and so we get penalized for that pretty heavily right we get a 2.3 penalty in the cross

[6:04:54] entropy which is which is contributed from this term right in the cross entropy calculation. So that should make some sense. um we So that should make some sense. um we get a very low error right here because

[6:05:08] in this like in this situation which aligns to what we just talked about we have most of our probability is on Samsung and that is the true label. There's a one in that first slot. So this is a much smaller contribution to

[6:05:22] this is a much smaller contribution to our error. There's still some error because it's only 90% probability. Ideally, we would have 100% probability on that and then our error would be zero, right? Because a logarithm of one

[6:05:35] is going to be zero. But we so we still have some error, but it's much smaller, right? Much smaller than the others. But all we have to do is basically uh take all we have to do is basically uh take all of these guys and average them

[6:05:50] together uh in order to produce the actual um cross entropy loss uh quantity. It's just an average. Okay, an average amongst these guys. Um, I think

[6:06:03] there's a typo here, by the way. I don't think it's 4.76. That seems kind of high. I think that's I think that's this total. And we have to divide that. We need to divide this by seven to get the actual average. Okay. Okay. So whenever

[6:06:18] actual average. Okay. Okay. So whenever we um whenever we do the binary case, the formula actually reduces uh quite a bit um into a simple uh a simple formula

[6:06:31] here that depends on the label. So when our label is one, we get this, which is just our because remember in a binary cross entropy, you're only going to produce a single probability. And what the intuition here is is the same as

[6:06:47] the intuition here is is the same as usual. Whenever this um whenever this probability is uh high, whenever it's close to one, that's a good result in to be pretty small because whenever

[6:07:00] that's close to one, this is a small value, right? including when it's all the way at one. If we had 100% probability, this logarithm would be zero. So we'd have no error. Um, so hopefully that makes sense. Whenever the

[6:07:13] hopefully that makes sense. Whenever the label is zero, we take one minus our probability. And so our contribution comes from one minus our probability. Uh

[6:07:25] comes from one minus our probability. Uh so if we're predicting uh like a 60% probability and the label is actually zero, that's a big mistake. Or like if if we're predicting a 90% probability and our label was actually a zero,

[6:07:38] that's a pretty big that's a huge mistake, right? Because in the binary mistake, right? Because in the binary case, remember the probability is just a single value that represents like the the class. If it's if it's above,

[6:07:53] usually our threshold is 0.5, right? If it's just above 0.5, we predict the one. If it's below 0.5, we predict a zero. So that's why we do one minus the probability when the when the class is zero. And so that helps us penalize the

[6:08:08] right thing. So if it's again like a 90% probability, but our label was actually zero, that's a significant penalty um because we would get logarithm of 0.1, which is going to be a big negative number. So that's why we have a one

[6:08:22] minus there. Now 1 minus 0 is just one. So this this just is always going to be So this this just is always going to be one in either case. But uh when the label is zero, we do one minus the probability inside this logarithm.

[6:08:36] But in in the same way we did for categorical cross entropy, we're still going to average that across all of our examples. So we still to compute our loss to compute our overall loss against all

[6:08:50] of our examples, we're still just going to average that for all of our data, right? To to get the average cross entropy loss. Okay, so that's really remains the same. Just a simplified formula a bit whenever we have a binary

[6:09:05] case because we're only producing one probability. Okay, so in code, this is kind of what it would look like. uh in in order to compute the uh binary

[6:09:17] cross entropy. Now you may be wondering in this code, you know, why do we have in this code, you know, why do we have uh why do we have this number here? And

[6:09:29] actually pretty typical in most implementations of cross entropy is they implementations of cross entropy is they have a small number here usually to avoid taking the logarithm of zero because the logarithm of exactly zero

[6:09:46] is negative infinity. So we don't want that. Um we never want to take the logarithm of exactly zero. So most under the hood implementations of cross entropy will safeguard against ever plugging in

[6:10:00] safeguard against ever plugging in exactly zero. And uh because we we don't want this to be exactly zero. So we'll kind of have this tiny tiny tiny amount to keep us away from zero. Uh if if just in case, right? Just in case to avoid

[6:10:14] in case, right? Just in case to avoid this. Now, it's still going to be a huge negative number. It's just not going to be infinity. Um, it would still be a be infinity. Um, it would still be a huge negative number if if we uh like

[6:10:27] let's say we predicted um a one probability, but it was actually zero class. That's a completely wrong answer. And we don't want to have infinite penalty for that. we still want a significant penalty which we would have

[6:10:41] um but we don't want an infinite um because that would just ruin the rest of the calculations right so that's why that's why that's that's why this is here okay but of course like we we aren't going to use this uh because

[6:10:55] pietorch and tensorflow have their own way of calculating this they have their own loss function that we'll be able to use this is just like if we were to codify the math that we just saw for especially

[6:11:08] binary cross country it would look like this. Okay. So the uh now inside of uh TensorFlow and PyTorch they have a reduced version of that formula that we

[6:11:23] saw earlier with the cross entropy and it looks like this. So the the one that we actually will use will be this version. Um, the reason that's the case is because remember the only contribution we get from every

[6:11:38] single term is there's a one sitting right in front of this, right? That's the only contribution we ever get is one times the logarithm of the probability. Um so instead of instead of writing out that formula like this where we do um y1

[6:11:55] times logarithm of um probability of uh probability of class one um you know uh and then we had plus

[6:12:09] um y 2 y 2 logarithm of probability of class 2 plus blah blah blah. We remember this

[6:12:24] formula is for the cross entropy the the in all of those cases the only time we get a contribution is is the term where the for the actual class right so uh if if the label had the second class then no other terms contribute this is the

[6:12:40] only guy that's going to contribute so for that reason we can actually simplify this formula to just this it just reduces to this formula here where we're taking the logarithm of that class um probability and we we technically have a

[6:12:56] one sitting in here sitting in front of it, but that doesn't really do anything, then we have our negative out front, which is this guy. And we have our average here, which is uh dividing by n and and totaling all these up. So, uh

[6:13:12] the formula makes a lot of sense. It's just um it just is uh reduced significantly from what we it's it's much simpler uh notation for writing

[6:13:24] that out than we had earlier. Okay, so this by the way this version of it is um often referred to as the sparse version or the sparse categorical cross entropy.

[6:13:36] Um you can use either one there. They give you the same result as categorical cross entropy. It's just a little bit simplified of a notation. That's really honestly the only difference. Uh that's that's the only difference is um you

[6:13:51] know we we just simplify the notation a bit from what we had earlier. This is sometimes called sparse. So we can use either um whenever we use pietorrch or tensorflow we can use categorical cross entropy or sparse categorical cross

[6:14:05] entropy. Either one gives you the same result. They're the same formula really. result. They're the same formula really. Okay. So the reason we use that is to um you know obviously call out mistakes in the probability. So if you're wondering

[6:14:19] like why don't we just use mean squared error or mean absolute error. Um it's because we don't want to calculate like a numerical distance that's not going to amplify those mistakes. What really is amplifying them is the logarithm that's

[6:14:33] heavily penalizing uh the mistakes. So, so that's why, you why wouldn't we just use the distance from the mean squared error because we kind of have almost like two vectors that we could calculate the distance

[6:14:48] between. Um, but that that's not going to emphasize the right penalty which is really a probabilistic penalty that we want to apply. So, so in that situation

[6:15:03] mean absolute error. we're going to resort to the cross entropy because it does what we want it to do as a loss function. It really penalizes those function. It really penalizes those wrong probabilities. It amplifies them

[6:15:16] correctly. So you do not want to use mean squared error. You can get uh bad training in that way. So so we want to avoid that and make sure we're always using cross entropy whenever we're doing classification.

[6:15:29] We should always be using that. um we should never be using mean squared error, mean absolute error in any classification example, right? It's just not going to penalize the right mistakes. Okay, so for just to recap for

[6:15:44] regression, we should be using mean squared error. That's fine. That's regression problems where we're predicting uh some type of continuous value that's coming out of our neural network. That's fine. for classification

[6:15:58] we should be using cross entropy loss. Um, and whether it's the sparse version or the regular categorical cross entropy or in the case of a binary problem, the formulas we should be using for the

[6:16:12] right? And in a classification, we should not be using mean squared error, do the right job and penalizing those wrong probabilities. Okay. So, I also

[6:16:24] wanted to call out that there is another loss function out there that's uh loss function out there that's uh sometimes used. I would say rarely used um versus cross entropy for classification and and it's known as the

[6:16:38] hinge loss. I'm not going to spend too much time on it cuz it's I've rarely ever seen it used. The the idea behind the hinge loss is to basically take the the hinge loss is to basically take the um take the cross entropy loss and and

[6:16:53] have some type of uh regularization to it almost like when we did lasso and ridge regression where we had this kind of additional penalty you know added on top of the original loss function of the mean squed error. If you guys remember

[6:17:08] all the way back in regression when we studied lasso and ridge, they added a penalty to just typical mean squared error, right? So hinge loss is going to be kind of that same idea. It's going to be a variation on cross entropy that

[6:17:24] be a variation on cross entropy that aims to kind of penalize uh penalize aims to kind of penalize uh penalize certain terms uh in the loss function. I'll show you I'll show you what it looks like. Um, but hinge loss also by

[6:17:37] looks like. Um, but hinge loss also by the way is inspired by the SVM. It's actually the loss function that's used in training an SVM uh in order to find that um hyper plane that separates the data um and increase the margin between

[6:17:54] the hyper plane and the points you know away from that hyper plane. So it it uh penalizes mistakes still in the same way that cross entropy does, but it also

[6:18:08] penalizes ones that are close to what we call kind of the the decision boundary call kind of the the decision boundary which is uh the sort of like an SVM like that um hyper plane that we're building to divide the classes. Um, so it just

[6:18:23] has this kind of extra penalty uh there that I I don't think is very natural for neural networks. I think it makes sense for SVMs,

[6:18:35] but it's not very natural for neural networks, which is why I think it's not networks, which is why I think it's not used as much as as cross entropy. Um, so used as much as as cross entropy. Um, so what is the hinge loss? It is the hinge

[6:18:48] loss is given by the maximum between 0 and then 1 - y * f ofx um which is the f and then 1 - y * f ofx um which is the f ofx is just a probability. So it's kind of a it's kind of an all or nothing loss in some ways too. Um it's it's also kind

[6:19:05] of a weird one in that it's very it's much harder to train using this because it doesn't have a very nice gradient. That's another issue with the hinge loss is this function is very ugly. It doesn't have a nice it doesn't have a

[6:19:20] doesn't have a nice it doesn't have a nice gradient at all. Um and the the reason it has this max is kind of an almost like a relu where you don't want to have anything negative. So your loss is um guaranteed to be positive uh in

[6:19:35] this situation and you don't want to have anything negative. But again, I because we're not ever really going to use it. We're gonna focus on cross entropy because that's that's what's used um I'd say 99 out of a hundred

[6:19:49] times I've seen I've rarely ever seen the hinge loss actually used on a especially on a neural network. It now that's not to say it's never used. It is machine under the hood to train it. Um it is the loss function that's used but

[6:20:06] uh it's not primarily useful. It's not that useful with neural networks. It hasn't shown to be that useful, but it could be used in theory. Okay, I'm just going to go through this. I'm not too worried about hinge loss, so I'm just

[6:20:21] going to go past it. So, one of the issues uh so one of the issues that I had mentioned earlier is it's it doesn't have a gradient. It's has a very um ugly kind of gradient. So it's for this reason it's it doesn't really perform

[6:20:37] well with training of neural networks because we really rely on that gradient because we really rely on that gradient uh in the gradient descent algorithm. So actually hurt the training process of a neural net more than it can help it

[6:20:50] functions and metrics and deep learning. Thanks for sharing that. So you can there's now there's other variations on the hinge hinge loss. are squared, which the hinge hinge loss. are squared, which can square this uh which attempts to

[6:21:03] make it better um and attempts to have slightly different penalties on mistakes and make it a little bit better in terms of the gradient, but still not nearly as good as what we see with cross entropy. So, it does like using the squared

[6:21:18] version helps a little bit for gradient descent, but it's still not nearly as descent, but it's still not nearly as good as the um as as the cross entropy. So we'll So bottom line is we will stick with cross entropy but just showing you

[6:21:30] that there are other loss functions out there that are sometimes used albeit rare rarely there's there's rarely ever a reason not to use cross entropy. It's a reason not to use cross entropy. It's it's so effective at at um pointing out

[6:21:44] probabilities. There's really no reason to use anything else unless you have a really good reason to. Okay. So to wrap up the lesson, we're going to just review forward and backward propagation and then talk a little bit about some

[6:21:58] regularization ideas which are a couple of ideas to help prevent overfitting just to remind us that you know overfitting is a huge issue especially with neural networks. Um so we'll just briefly talk about that towards the end

[6:22:12] and we'll see those ideas come back up when we start building our own neural networks especially. Um but let's talk about forward propagation just to remind about forward propagation just to remind us of what that really means. So forward

[6:22:24] propagation really uh revolves around sending data forward through the network uh in order to make a prediction. So whenever you hear that word forward prop or forward propagation, uh that's just referring to the idea that data kind of

[6:22:40] propagates forward from layer to layer to layer to layer. And we know what's actually happening there on every layer is really what we're doing is taking a weighted sum passing it through an activation. That output goes on to the

[6:22:54] to become the input to the next layer and on and on and on until we finally reach the output. And so this is just the forward propagation. Now when we have more advanced neural networks, there may be times where there's

[6:23:07] connections that actually go backwards. And so not everything is forward And the prime example of that's going to be an RNN when we get to that later. But right now, um, data kind of propagates forward for us always in the, uh, kinds

[6:23:23] of layers that we've seen so far. mainly just awaited sum through an activation and that just goes forward through the network to ultimately to to produce some

[6:23:35] type of output. And remember the output is going to be different depending on the network. It could be a single value. If it's like a binary um classification, it could produce a single probability. It could even be uh a regression, right?

[6:23:50] So if it's a single output like this, it could be a regression might be a binary classification. Um or we might have more than one output node or I should say than one output node or I should say neuron which would be like in a um in a

[6:24:04] multiclass classification. We might have three output neurons or 10 or a thousand depends on how many classes we have we're predicting. And remember those middle layers there are typically called the hidden layers of the network. We

[6:24:17] usually have our input layer up front which matches the size of the input. So if we have kind of n features have n input nodes and that just orients our data to kind of start flowing forward and our hidden layers are going to be

[6:24:32] configurable. So of course we're going to practice this today like building out these hidden layers especially in TensorFlow. Um, but that's something kind of setting up and architecting these hidden layers and architecting the

[6:24:47] output. Um, of course, as I said, the output depends on the problem if it's regression or classification. Okay, so forward propagation is a good analogy to it if you're if you uh if you think about it is kind of like

[6:25:04] our linear regression, right? in linear regression it's and this is a very very basic example of it with one feature and and kind of one uh weight and one bias. So in this situation like this forward propagation is just putting in our data

[6:25:21] and computing the output from the line which is computing this y that's what we're doing in forward propagation of the network. So what we imagine for the neural network is this is much more complicated. it's awaited sum through

[6:25:37] inactivation over and over and over again. So it's repeated um it's repeated quantities of this over and over and over again in our layers. Um so it's

[6:25:49] much more complicated, but ultimately it's the same kind of idea that we had in linear regression. We're just sending a data point in and getting an output back. And it's this is kind of like in linear regression that's kind of like

[6:26:02] forward propagating. we just put in an input and it it generates an output. Same thing with with forward propagation. Um we're generating an output. It's just obviously the model is much more complex. There's more layers.

[6:26:14] There's more computations going on than just MX plusb, right? Just a basic linear regression. Okay. So I wanted to show this demo um that kind of goes through forward propagation and and just reminds us of sending data through these

[6:26:32] reminds us of sending data through these layers. Um so if you go into lesson 4, it should be the only notebook there. So so you want to pull that up. I will pull that up as well. All right. So it should be this notebook. I just wanted to go

[6:26:45] through this to show us um just remind us of forward propagation and kind of how data flows uh throughout um the layers. So hopefully you have this demo. This is again in lesson four. It should be the only demo there.

[6:26:59] be the only demo there. Okay. So we're going to use uh numpy here and uh maybe some plotting. So we have and uh maybe some plotting. So we have piplot uh plt from mapplot lib.

[6:27:11] Pretty standard. Um now we're going to generate some random weights. So remember this is what happens typically in a um typically in a neural network

[6:27:24] whenever we initialize that network it's going to have random weights to begin with. So this is not out of the ordinary for a neural net to kind of start with random weights. That's what's going to happen. Um, now this is mimicking this

[6:27:38] 11 here is just it's nothing special, but it's just saying we have 11 elements. So this this would be like we have 11 features and we have 11 weights and one bias. So this would kind of be like a linear regression that has that

[6:27:52] like a linear regression that has that has these 11 uh 11 features um 11 weights and one bias. So just kind of like that initialize those random values. So if you look at this weights uh array, it's going to have 11

[6:28:07] elements. The bias has one element and the inputs have 11 elements as well. Now we could take a look at the relu activation. So remember the relu

[6:28:20] activation uh takes whatever input comes into it and it computes the maximum between zero and that input. This is the rel very very basic nonlinear function

[6:28:32] return whatever it was as long as that's positive. Very simple function. So we're just defining that. Now hopefully this looks familiar here. So forward propagation for a single

[6:28:47] So forward propagation for a single layer. So for a this is really um mimicking a single layer in a neural net because

[6:28:59] if you think about it what's happening is we are taking the the sum of the weights times the inputs which is what the dot product does right it's the sum of the weights times the inputs plus the bias. So we take that weighted sum and

[6:29:15] that gives us this and we put that through the activation. So that value is then going into our relu and producing an activated an activated value which is the output of this layer.

[6:29:30] value which is the output of this layer. So this is our forward propagation. We get this value. So we are forward propagating this data through this single very very simple layer which is just um has some weights and a bias and

[6:29:46] then we pass that through a relu activation. Um now when we do tensorflow this when we build out our network going to be different but this is the main idea under the hood of what we're doing. So hopefully this jogs our memory of

[6:30:00] what forward propagation is doing in the code. Weighted sum through an activation. We see that again.

[6:30:13] here. This would be like computing our loss. So if we come down here um if the loss. So if we come down here um if the target was one we could compute the uh the the mean absolute error which is kind of what this is doing is taking the

[6:30:28] absolute value between the prediction and the y. So this is um mimicking the mean absolute error loss function. That's all that this is doing because it's taking the absolute value and we

[6:30:43] would total that up. But there's there's really only one entry in this. So the sum is just the uh it's just the single value. Um it's not really there's only one error. But obviously if we were doing mean

[6:30:57] and then divide by the number that there are to average it. Actually, we wouldn't even take the sum. We would likely just call the mean. Just do the mean to get the average of the errors. Um but in this case, there's really only there's

[6:31:10] only one entry in the error. So this is our mean absolute error for the loss and we we'd be off and running with that like we could use that to go back and update our weights with a gradient descent update um all of that. So so

[6:31:26] just reminding us of kind of a typical forward propagation compute loss back propagate right that's what we would actually be doing in in reality.

[6:31:41] instead of absolute value we could compute the square that that'd be another option for our loss that we could use for this kind of regression could use for this kind of regression example. Um and so the uh sum of squared

[6:31:54] example. Um and so the uh sum of squared error is uh 2.47. what forward propagation is doing, which is mainly this idea weighted sum through

[6:32:09] an activation. And when we have a larger network, um you know, we're going to that that would be occurring in many many layers potentially, right? This this weighted sum through an activation and it would be in this network. It

[6:32:22] would be kind of managed doing that process. Okay, so let's remind ourselves of back. So forward propagation is just sending data forward weighted sum through activations mostly. Let's remind ourselves of back propagation which is

[6:32:34] ourselves of back propagation which is going to be uh remember it is our way of going back and updating weights throughout the network based on the loss that we calculate and particularly the gradient of that loss because we there's

[6:32:49] a gradient descent formula that dictates how every single weight should be adjusted. And remember on Monday we did that very very detailed calculation which I encourage you guys to go back and kind of look through uh because I

[6:33:04] think it's a really good example that goes step by step. Um and I was saying the math isn't that important in terms of those gradients. I don't think it is but I think the process is really important to

[6:33:18] understand that we are you know going backwards in the uh layers to uh we're going backwards in order to adjust all these weights right we start from the

[6:33:31] towards the front updating all those weights so back propagation is really important for training it's not important for predicting so once it's

[6:33:43] trained we will never do back propagation unless we are going back and retraining. So back propagation only occurs during the training process. Once a model has been trained, we only are doing forward propagation, right?

[6:33:56] Because we're producing predictions. So really only doing back propagation during the training process. And hopefully that makes sense because the training process is is what is adjusting all those weights. So of course in order

[6:34:11] to do back propagation we need to compute the error. So once we've done forward propagation we can compute the error based on our loss function that we error based on our loss function that we have. And of course the the the the

[6:34:24] typical thing we use in the loss function is going to be whatever our propagation perspective whatever our neural network produced and the actual label. So this is where I said last time we really need the actual label. This is

[6:34:39] still a supervised learning problem. Really need those labels in order to compute our loss. Really need that loss in order to update all of our weights and adjust our network to actually learn anything meaningful.

[6:34:52] Right? We need this loss. Um so what we just learned is um we should be using cross entropy in a classification problem. we should be using something

[6:35:04] like mean squared error in a regression problem for our loss. And of course, we'll be able to set all that up inside of TensorFlow or PyTorch. They they of TensorFlow or PyTorch. They they manage that for us. So um remember the

[6:35:17] manage that for us. So um remember the back propagation is utilizing um or is being utilized by gradient descent because the gradient descent gives us the formula for what the weight update should be during the back propagation

[6:35:31] process. And if you guys remember um I drew this picture which I think is a really powerful picture is is this kind of idea that if we have this is a graph of our loss um loss function. Um what the gradient

[6:35:47] descent is really doing is finding out where we are on this which is our loss value. So we know because we can compute the loss, we know where we are on this

[6:35:59] and we're taking steps towards the minimum, right? We should be taking these kind of small steps down this way towards the minimum of the loss down here, which once we get to that point, we have a very good model at that at

[6:36:13] that point, right? basically have no error um at or minimal error you know at the at the loss uh minimum back propagation though is going to take every one of these updates. So when we take a step and we know like when we

[6:36:29] take a step that the step is going to tell us the formula for how we should adjust the weights um and and really we need to utilize those calculations um

[6:36:41] those weights and that's what back propagation is going to do is propagate propagation is going to do is propagate those weight updates from back to front according to this uh gradient descent uh formula here.

[6:36:55] So gradient descent plays a really important role in computing what our weight update should be and then back propagation propagates those weight updates from the back to the front. Okay. And then we had a picture like

[6:37:09] this. Remember, I drew a picture kind of like this earlier on Monday where we looked at um the back propagation really relying on forward propagation in order to produce uh the quantity of what do we have.

[6:37:24] have. Uh and then we we compare what we have to label and then we use that that is what helps us compute the loss. Ultimately the loss

[6:37:37] is what influences the gradient descent update, right? And that update gets back propagated throughout the network to to adjust the weight. So you can think about like training a neural net. I think a really good analogy of it is

[6:37:51] kind of like we have a million little knobs and we're coming up with a formula that's telling us how much should I turn every little knob. And back propagation is saying what I should do is start with the knobs that are closest to the end.

[6:38:06] Start with those knobs. Turn those however much I need to. Those knobs helps us compute how much I should be turning earlier knobs, right, that are in the front of the network. And so then by the time I do one training iteration,

[6:38:21] I've made all the little tweaks to my knobs that I need. And now I'm set up the next time around to hopefully do better. That's that's the idea for the better. That's that's the idea for the training. So, you know, ideally when we

[6:38:33] make an adjustment, that's kind of what this slide is saying. When we make an adjustment in back propagation, um, we should be doing better the next time. And what we should see visually, like if we were to plot the loss

[6:38:47] over the course of the training, um, we should see that decreasing. things should be getting better over the course of our iterations, right? So maybe the loss starts somewhere up here. It should be getting better and ideally

[6:39:02] um flattening out to kind of a minimum uh as we go through the course of the iterations. What we do not want to see is increases, right? We don't want it we don't want it to kind of do this. That would be bad. We don't want that to

[6:39:15] happen because then that means that it's not really training. It's not really learning anything at that point. Um, we want it to keep decreasing. And something that we're going to talk about coming up shortly is we really want to

[6:39:28] be cautious about this is like yes we want it to be decreasing on the training want it to be decreasing on the training data but ideally if we like once we make all these adjustments something we can do we haven't really talked about this

[6:39:41] that much is that some you know when we make these adjustments we can apply that network to test data that's been held out and we can generate an equivalent graph to say okay what is our loss function on test data and ideally that's

[6:39:57] also decreasing. That would be nice. Like so this blue line is is the loss that's generated on a hold out set of test data. And that's that's actually going to be one of our strategies is to do a train

[6:40:13] test split on our data just like we did in machine learning and then evaluate our performance on the test set kind of simultaneously. So we so we train on the training set, make all of our back propagation changes on every iteration.

[6:40:28] What we should be able to do is apply that new network, the adjusted values of the weights, the network on both sets. We should be able to apply it on the training set. We should also be able to apply that on the test set and see what

[6:40:42] apply that on the test set and see what our performance is. Ideally, we see a the same kind of trend. we see the loss decreasing on that test set um over the getting better and better and better as well. What would be bad is a a

[6:40:59] divergence of these two. So what I mean by that is um if we drew a graph like this, let me go back and redraw this. So if we had our loss

[6:41:17] what we do not want to see is a decrease. So this would be on the training data and then what we don't want to see is a

[6:41:30] divergence on the test data. So in other words, if we apply this to the test data, it kind of tracks initially and then kind of diverges away. This this is kind of a a difference that's significant difference there, right? We

[6:41:45] don't want to see that because that would signal an overfitting

[6:41:59] why is it overfitting? because on the training data, our loss is decreasing very steadily and it actually gets pretty low um over the course of iterations. That's great, but there's a there's a big difference on our test

[6:42:13] data that signals that we're overfitting to our training data and not able to generalize our network to test data because our loss is much higher. So that divergence is a very clear picture of overfitting. So that's something we're

[6:42:27] going to be on the lookout for. What's really cool is like TensorFlow in particular actually gives you the ability to generate this plot during the course of the training. So, so every iteration it can update this plot and we

[6:42:40] can see it in real time how the training is progressing on these on these two sets of data and that way we can kind of see that divergence occurring and then keep training because I I've already seen enough on the overfitting this

[6:42:55] seen enough on the overfitting this divergence. So we'll we'll see that when we get into TensorFlow is um there is that ability to kind of uh visualize our training process which is pretty pretty helpful. Uh if it's underfitting what do

[6:43:11] you think what would be a good what would be a sign of underfitting? High. Yeah, it should be high because what does what does the loss function track? does what does the loss function track? It it tracks our mistakes. it it gives

[6:43:24] us a distance which is a distance between the actual label and our predictions. So the higher this distance the more error we have. So the picture the more error we have. So the picture for un for um underfitting would look

[6:43:38] something like this. Yeah it the basically for underfitting it would let me try to draw it alongside this graph. It would be let me do it in purple. So

[6:43:50] It would be let me do it in purple. So um for underfitting it might um be it might be much higher kind of like this. It basically never decreases. It basically never decreases and stays up way too high. The loss is just too high

[6:44:04] in general. Um this would be this would be kind of uh underfitting that would that would be underfitting as well. It's just not performing very

[6:44:20] well. Are our losses either increasing or staying kind of stagnant and it's too high. Uh [clears throat] if we add a hidden layer is there any impact on this graph? There might be. Yeah. So, um, if we add a hidden layer, uh, there's

[6:44:35] there's no direct way to know if it's going to help us on this graph, but, um, that if we are underfitting, remember what underfitting kind of signals, it signals that our model is too basic. So, if we're underfitting, one of the things

[6:44:50] if we're underfitting, one of the things we can do is maybe add a hidden layer. do if we're underfitting because maybe we need a more complex model. Okay. Now, on the reverse side, if we're overfitting, maybe our model is too

[6:45:04] complex and we actually want to remove a hidden layer. We have too deep of a network. That could be the case, too. Does that make sense? So, like underfitting and overfitting tell us maybe what kind of adjustment we should

[6:45:16] make to our network. Should we remove a layer? Should we make the layer smaller and less neurons? You know, those things we can kind of tell from a graph like this. All right, let's talk about regularization. So,

[6:45:31] we're kind of talking about that now in terms of overfitting and underfitting. Um, and that's really what regularization is getting at. We talked about regularization actually a little bit in regression. Remember,

[6:45:43] regularization um was involved us doing something like lasso or ridge regression where we added penalties um to combat overfitting.

[6:45:55] So with neural networks, you know, we can also talk about regularization because neural networks actually have a tendency to overfit. They are very very prone to overfitting because they are

[6:46:10] remember things that are very complex tend to overfit and neural nets are neural networks are very complex. They can be very complex with many many layers and many neurons. And so they can be easily susceptible to overfitting.

[6:46:26] And so we have a picture kind of like this um where we uh I just drew something exactly like this. So, you know, we can definitely overfit um in in

[6:46:38] a situation like this where we uh uh maybe on our training data we have a picture like this, but then on our test data it we're kind of tracking with it and then we kind of diverge would be um certainly an overfitting situation.

[6:46:56] um because there's a there's a huge gap here that's just growing and growing and growing between our training performance and our test data performance in terms of the loss. So that's overfitting and um this can easily happen if you

[6:47:12] think about also the training process of a neural network. There's usually many iterations and every iteration we're sending all of the data forward in order to generate the the average loss which updates all the weights. And and if you

[6:47:27] think about it, we're making all these adjustments to all these weights. we're doing it so many times in these iterations means that the neural net just the training process makes it potentially

[6:47:43] susceptible to overfitting just just relatively easil easily to overfit. It's such a complex model. There's so many iterations. Every iteration seeing all the data, it it means it can overfit really easily to training data and and

[6:47:56] struggle to generalize to new data such as a test set or in worst case scenario like actual predictions that we need to make on real life data, right? So it can overfit pretty easily. Um we're going to discuss a couple of ways to combat it.

[6:48:14] discuss a couple of ways to combat it. One way to combat it that is shown here on this slide is what is known as early stopping. So early stopping is a is a stopping. So early stopping is a is a very interesting uh idea that um PyTorch

[6:48:28] and Centrafo both have an early stopping algorithm. And essentially what early stopping will do is keep track of the loss on the test and the training and if

[6:48:41] it sees a large enough divergence meaning there's a divergence here that's greater than some threshold. So if this distance So if this distance between the uh training loss if the

[6:48:54] distance there becomes greater than some threshold let's call it alpha if this distance becomes big enough then we kind of trigger a stop basically to

[6:49:06] say I don't want to continue because my my my uh test um evaluation of loss is starting to diverge away from my training. So, I should just stop it

[6:49:19] here and stop training here and and maybe that's good enough. Um or I should like I stop there so I can make some adjustments. I don't want to keep going. So, um there is this is actually built into TensorFlow and PyTorch. Uh and it's

[6:49:35] called early stopping and and you're able to set that up. Another thing you're able to set up with early stopping it is basically like a grace stopping it is basically like a grace period to say, okay, if I if I see this

[6:49:47] distance greater than some threshold, I'm going to trigger an early stop. If I don't see an improvement in the next uh x number of steps, so as we go through the next 10 iterations, if I don't see an improvement and I

[6:50:03] don't see this distance drop down back below alpha, yourself kind of a grace period there too in the early stopping to give it a chance to recover. Maybe it's just bad. It gets unlucky and it's just bad on

[6:50:17] that iteration and I make some adjustments or the training adjusts the weights and then it starts to do better on the test set and it goes back below that alpha and in the next couple steps and therefore we've recovered and

[6:50:30] everything's okay. So so you can give yourself some grace there. uh and and that's all configurable in the early stopping algorithm and we're actually get in lesson five when we do TensorFlow. Um so so that is one thing

[6:50:46] we can do to combat overfitting is to just stop our training early. We don't want to let it keep going if it's performing poorly on test data. Don't let it keep going. That's basically the the simple answer there. Just stop it.

[6:51:01] um and and either stop it and use that model from the point you've stopped or um try again and maybe make some adjustments to your to your architecture. Uh maybe remove a layer, lower the number of neurons, something

[6:51:15] like that and try again. So early stopping, we're going to see that uh stopping, we're going to see that uh later. Um so that's one way to combat overfitting is to just stop the training early. A couple of others that are

[6:51:28] early. A couple of others that are widely used are what are called um L2 regularization. Um and and both of these uh are basically ideas that are help that are used to to mitigate against overfitting.

[6:51:45] used to to mitigate against overfitting. Now this one we've actually seen before Now this one we've actually seen before with ridge regression. So this is very with ridge regression. So this is very analogous to uh ridge regression and

[6:51:57] remember when we did lasso and ridge regression this is actually the same idea is we're going to introduce a penalty in the loss function penalty in the loss function um in order to penalize large weights um

[6:52:11] which is supposed to help prevent overfitting. So it's kind of the same exact idea there in like when we did ridge and and regular linear regression ridge and and regular linear regression is we wanted to discourage any one

[6:52:23] weight from dominating. So we penalize very large weights. Um we can actually do the same thing in neural net training and penalize large weights um or any one weight from being dominant with a similar kind of penalty term in

[6:52:39] the loss function. That's one way to do it. And we're going to talk about dropout regularization which is a whole different concept. This this is actually a very interesting concept that is widely used in neural net training

[6:52:53] way more than L2 regularization. So so dropout is widely used and we'll actually use it quite a bit uh in our examples uh especially when we get into TensorFlow. So I'm going to talk about dropout regularization coming up and

[6:53:07] tell you more about what that is but it is definitely widely used. uh L2 regularization not quite used as much but just something to be aware of that we could do is just add basically adjust our loss function to add to to prevent

[6:53:21] overfitting. So what does that mean? We basically have um some type of penalty term that we add to our existing loss function. So in mathematical terms, this

[6:53:35] function. So in mathematical terms, this uh C 0 is just our initial cost or loss function. Whatever we had, we still have, but we add in this penalty have, but we add in this penalty that discourages um large weights. So it

[6:53:48] penalizes weights with a squared penalty there um to to kind of discourage those. So when you're training, you're going to get an extra penalty if you have a really large weight. And so it so this helps make the weight adjustments and

[6:54:04] control those so they're not overly you know overfit to the training data that it that it's being trained on. So it's the same idea we had. This formula might look a little familiar. It's because it is very familiar from

[6:54:18] the uh ridge regression. It's exactly the same idea that we had there. Um and was just mean squared error. But here you know in classification it could be

[6:54:31] it could be cross entropy in regression it could be mean squared error again. So same kind of penalization that we're doing that we we had back then. Okay. So this is an option and it's actually not too bad to set this up in PyTorch

[6:54:45] TensorFlow to take an existing loss function and just make an adjustment to function and just make an adjustment to it. Um most of them have a regular regularized version of the existing loss functions like cross entropy and mean

[6:54:59] squared error um that you can just grab out of the box and use. One thing to be out of the box and use. One thing to be aware of is if you do use this basically uh it can take longer to train because it's it's has because it has this extra

[6:55:14] penalty the gradient is a little bit more complex and so the back propagation is going to take a little bit longer. So the training can take longer with this regularization in there. That's the only kind of caution. Okay. So that's one

[6:55:29] kind of caution. Okay. So that's one approach is to add a penalty to our loss function which might make the training take longer but it's designed to discourage large weights and prevent overfitting just like we saw with ridge

[6:55:42] regression. So that's one option. Now another option that is widely used is a another option that is widely used is a very very interesting idea is randomly getting rid of neurons

[6:55:56] during the training process um on every iteration. So what you can imagine is that each iteration of the training I basically select neurons and with a certain probability I'm going to turn them off. So it's as if they're not

[6:56:12] there. Now, doing that has a significant effect on the training because on that effect on the training because on that iteration, I'm going to basically have a much different architecture if I delete a bunch of neurons. Um, and then maybe

[6:56:28] different ones and then on the next iteration I delete different ones. So, it's not like they're permanently deleted. It's just during that iteration they're not going to be involved and

[6:56:42] involved during every training iteration. And this is a very clever approach because essentially what happens during because essentially what happens during the training is your model learns how to

[6:56:56] how to adjust the weights across all of the iterations um with some of the network turned off on every iteration. And the idea behind that is to prevent overfitting because you don't have a full network with all connections on

[6:57:12] every single iteration and that basically prevents having over complexity in your model. It's a very interesting approach. So again just to just to reiterate what dropout is doing is it is drop the reason it's called

[6:57:27] dropout is because the the deletion of the neuron is actually called dropping the neuron is actually called dropping it out. So if I had a network like this and it let's say it produces one output and so there's um obviously a bunch of

[6:57:42] connections in this network. Um so it'd be you know in this network. Um so it'd be you know kind of like this.

[6:57:56] and then connections like this. Um, essentially what we're going to do is on every iteration we choose with a certain probability to drop out a a neuron or probability to drop out a a neuron or multiple neurons from this uh from this

[6:58:12] uh from each layer from each hidden layer. So let's say we on this hidden layer like on the first pass we might drop out this neuron with a certain probability right it it has a chance to be kept and that's the probability of

[6:58:26] keeping the neuron is is actually a hyperparameter that you have to choose whenever you use drop dropout and it it has an effect like if you were to lower that probability to be something really low

[6:58:41] low um then you're basically um you're underfeit um which which could be bad. If you keep everything like if you have a really high probability then basically you're

[6:58:56] going to ensure all your neurons are kept which uh basically will potentially lead to overfitting anyways. So you want that probability to be somewhere in the middle. You want a decent chance of being dropped out, but

[6:59:11] not um overly low or overly high. And remember, if it's overly low, then you're going to drop out way too much and have an underfit model, a very simple model, because you're going to just delete so many neurons. Um you're

[6:59:23] effectively. So, that's kind of the worst case is you like drop out everything and all of a sudden you're not learning anything at all. You don't have any any neurons to use. So, you have a very underfit model in that case.

[6:59:37] So you don't want that to happen. So you want this probability to be ideally like want this probability to be ideally like something like 0.4 to 0.6 somewhere in that like middle range maybe 0.5. You want it to be somewhere in there. But it

[6:59:50] is something you can change during the training. You could play around with it and hyperparameter tune. Try out different probabilities for the dropout. Now one thing I should also clarify is dropout is not permanent. So it when

[7:00:05] this is only during the training that the neurons get dropped out and it it they're not permanently going to be deleted because on the next iteration we

[7:00:17] deleted because on the next iteration we may keep this guy and then turn off this guy and keep this guy. So it's every iteration that probability resets and you have a a new chance to drop things out on each iteration.

[7:00:31] And then when the model is done, you're not dropping out in prediction. So in network. Like when you do a forward propagation and you're sending data through to make a prediction, you're not going to turn off things randomly during

[7:00:45] going to turn off things randomly during prediction. This is only used during training. Only used during training. And it's the purpose of it is to prevent overfitting. It's a very clever approach to preventing overfitting is to

[7:00:57] basically artificially kind of change our complexity of the model by turning interesting approach. But one of the things we're going to see is setting up

[7:01:09] dropout in our network is a really easy process. It's it's not like in terms of training. It's not trial and error. Essentially, we'll just set up the dropout layer um and have the ability to drop out any layer that we want. So, we

[7:01:24] could we could set up and say I want to drop out this layer, but you know, of course, I don't want to drop out the output. We never want to drop out the input. I don't want those to be cut off at all. I only want to drop out the

[7:01:38] hidden layers. So, we should only be doing this to hidden layers, right? I should not be dropping out the output and certainly not the output and I and certainly not the output and I shouldn't be dropping out the input.

[7:01:53] doing this to? Okay. So here's a good picture better than what I drew is like if you're um dropping out things and again the whole goal is to do this so again the whole goal is to do this so your model generalizes better by uh

[7:02:07] during the training process by kind of strengthening its learning ability. It's not it's not getting overfit by having everything activated all the time. It's everything activated all the time. It's kind of learning um on the fly by, you

[7:02:22] know, getting stronger in different areas by turning off certain neurons every single iteration. Um you get a picture that's kind of like this. When you turn off a neuron, you are severing their connections, right?

[7:02:36] you go from something that's really complex, which is the whole point of of preventing overfitting, right? You go from something really complex from something really complex to something simpler.

[7:02:51] That's the idea. But we're we're not going to keep it simple permanently. So on the next iteration, it might look slightly different because we would drop out different neurons, right? And so we're basically going to do this over

[7:03:06] and over and over again. And it's a good method. It turns out to be a really effective method to prevent overfitting by going and the idea is just to kind of keep alternating from complex to simple complex to simple and that way you kind

[7:03:21] of distribute your learning very effectively which is supposed to make it generalize. Okay, does that make sense on the overfitting or the prevention of overfitting like how it's preventing

[7:03:34] overfitting? It's it's going from complex to simple and really alternating that over the course of the training by it's it's all random. So, we randomly choose which ones to to turn off and on. I really want to emphasize that it's not

[7:03:49] permanent cuz like it could be like this on one iteration and then the next iteration um completely different ones are turned off. And once the model has been trained, nothing will get dropped out. will use the full model to make a

[7:04:05] prediction. It's just the what's the idea is the training is supposed to prevent the overfitting so that by the time we get the full model um it it has a better chance of generalizing because it it's basically been taking on these

[7:04:20] different forms and and learning in different ways the whole time during the during the training process. So yeah, drop this is called dropout. Dropout is very popular. It it um came around in the late 2000s like early 2010s I think

[7:04:36] the late 2000s like early 2010s I think it was invented and um it's been widely used ever since in most uh in most training of any neural net uh will use dropout to some extent. It's a very popular approach to prevent overfitting.

[7:04:51] And again overfitting is a very prominent issue in neural nets. they tend to overfit very easily because they're so complex of a model. This helps eliminate complexity, right? It makes the model simpler every

[7:05:06] iteration is a very very neat approach. So again, one of the uh things to be So again, one of the uh things to be aware of is that probability of aware of is that probability of retaining a neuron. So that is known as

[7:05:20] retaining a neuron. So that is known as the parameter p. And so when you have a very low P, meaning you have a very low probability of keeping a neuron, so most of it gets dropped out, you have underfitting.

[7:05:34] That makes sense. So if we go back to this picture, if we had a low P, basically everything's being turned off. So you have a ve you have an overly

[7:05:46] simple model. Overly simple. So you're underfitting, right? So you you don't want to do that. And so this this um graph on the next slide kind of shows that happening. Whenever they had a very low probability of being kept, they

[7:06:02] underfit, which is what this is showing. There's a high rate of error. The the the loss function is pretty high on the training and the test set. They're underfitting. Now, as they increased the

[7:06:16] P, which is going this way, and they got more into uh this realm, they got more into this realm, um they saw better results. So,

[7:06:28] they had low training error, low test error. It's still like not perfect. There's still some level of overfitting here because there's a gap between the two, but it was better. It wasn't as underfit as in this this area here. So

[7:06:43] that middle ground is generally a good a good spot to be with a you know kind of a 50-50 chance of turning off a neuron. Um seems to be a good a good Um seems to be a good a good probability. Now as they kept going

[7:06:58] the overfitting got drastic more drastic right because what this means when the P is really high in this range is you have a really high chance of keeping the every neuron. So you're basically learning on every iteration with a full

[7:07:14] model. You're not turning anything off, which is keeping it overly complex, which is keeping it overly complex, right? So when you have a very high P, turned on. It has a very high chance of being turned on, which means you have a

[7:07:30] very comp you basically have this, which is a complex more complex model. So you're more at risk of overfitting in those situations. So that's kind of what they discovered. Um, which makes a lot of sense intuitively. Like this this

[7:07:43] just kind of verifies what we would think would happen. Um, where you get you get more overfitting the the further to the right you go on those probabilities. More overfitting. So very interesting. And by the way, this number

[7:07:58] so if you're reading this slide, this number refers to the size of the uh number refers to the size of the uh layers. So it's 784 because the images

[7:08:10] on the mnest data set. So if you guys remember imnest is the handwritten digits, right? Those are the images of things like a two or a one or a zero or a nine or whatever, right? Those are those are those images. So there's the

[7:08:23] look at what the output layer is. It's at the very end here is 10. That makes sense because it's the digits 0 to 9. So there's 10 different classes that we

[7:08:35] could predict. So the output layer needs to be a 10. That hopefully resonates and makes sense. Um, and the input layer is 784 because it's a 28x 28 image

[7:08:47] 784 because it's a 28x 28 image resolution. So 28x 28 is 784. resolution. So 28x 28 is 784. So um, the input has 784 pixels in total So um, the input has 784 pixels in total that are in this 28x 28 image. Um, and

[7:09:01] then they have 24,48 neurons in this layer. 24048 neurons in this layer, 24,48 neurons in this layer and then 10 output neurons for the classes 0 to9.

[7:09:13] output neurons for the classes 0 to9. Okay, so on that architecture is what they applied the dropout to these layers and came up with these numbers. Okay, so that looks like this. We will build a model that looks almost exactly like

[7:09:28] this architecture. We'll we'll practice that on the MNEST data set with those that on the MNEST data set with those images. Okay, so uh a lot of ground that we covered there in lesson four, but that pretty much wraps up lesson four.

[7:09:41] Um we talked about deep neural nets and how to do forward propagation uh back propagation to update all the weights. We talked about regularization which is We talked about regularization which is really um helpful especially dropout um

[7:09:55] to basically turn off neurons to prevent overfitting. We also talked about loss functions which are really critical for uh doing uh the training process, right? We need to know what the loss function is um so that we can evaluate the error

[7:10:13] is um so that we can evaluate the error uh that is um very critical in making that gradient descent update for the network. Okay. So, if you want to pull on to lesson five, which is going to be the first of two lessons that really get

[7:10:27] into our two frameworks for building neural networks, which are going to be TensorFlow and PyTorch. So, this lesson will cover TensorFlow and lesson six will cover PyTorch. Um, and of course along the way we're going to do examples

[7:10:41] with both so we can start to see how we actually build um, neural networks and see what goes into the training process and building in the code right inside of our Python. So that'll be pretty exciting I think. Um, so by the end of

[7:10:57] this lesson um, we want to see the main thing we want to do is get some hands-on experience with TensorFlow. So this lesson actually has a lot of slides that are with some different aspects of TensorFlow that we won't actually be

[7:11:11] touching. Um like TensorFlow playground and TF learn won't actually be touching those too much. So in the slides you're going to see me skip around a little bit because I want to get to the parts that we are definitely going to use which are

[7:11:25] mainly going to be TensorFlow and the underlying uh or the uh library that kind of goes on top of it which is caris um as kind of the interface to

[7:11:38] tensorflow to help us build neural networks. Um so that's where we're going to focus our attention. I know there are other slides in here about different um aspects of TensorFlow but we won't actually be using those really for

[7:11:52] anything and so we'll be using just caris mostly to practice building our nets and training them. So that's where we're going to focus our attention in the lesson. Okay. So let's just get a basic introduction to uh TensorFlow

[7:12:07] itself. So if you guys remember um we talked about TensorFlow very at the very very beginning of this journey where we talked about um you know these frameworks being developed to help um build and train neural nets and

[7:12:21] build and train neural nets and TensorFlow is one of them and so uh it is an open-source Python framework for building uh neural nets um and it was developed by Google uh back in the day to help them um to help them build and

[7:12:39] train neural nets. And it was open- sourced in 2015, so not that long ago. sourced in 2015, so not that long ago. Um but you know, by now over 10 years ago, um it has been kind of open sourced and available for free use. Um now,

[7:12:56] TensorFlow itself is an remarkable library. It basically handles all of that computation of forward and backward propagation. Especially that backward propagation that we saw that was so tedious, right? Computing all those

[7:13:11] gradients and then multiplying all those pieces together to generate a weight update is very tedious and hard to keep track of even manually. But what track of even manually. But what TensorFlow does is actually uh keep

[7:13:26] track of that very effectively in kind of a graph structure. Um so so TensorFlow is really a library that manages the data graph and data flow of neural network data which is um primarily going

[7:13:44] to be tensors for us which are basically like matrices. Um so it's it's this really remarkable library to manipulate that data in this kind of graph network that data in this kind of graph network format um which mimics you know a neural

[7:13:59] format um which mimics you know a neural net. Um so TensorFlow has a bunch of stuff that comes with it. Obviously the core library is something we'll use mainly actually mainly the library that's built on top of it which is caris

[7:14:13] tensorflow but it also has surrounding tools and other community resources such as documentation that's really amazing um that we will look at from time to time as we need to. Um, one of the tools that TensorFlow has that we'll look at

[7:14:29] later is called Tensor Board, which is a really amazing tool to kind of visualize that training process. So the the kind of graphs that I mentioned where you can kind of see your your loss um over the course of the training iterations uh is

[7:14:45] something that tensorboard can do which is a tool that uh was created by um you know the folks who made TensorFlow and they have many other tools like that. they have tools for like serving models uh um like in terms of deploying them

[7:15:02] and making them usable through like uh APIs and things like that. So have a lot we'll focus our attention on primarily the libraries that enable us to build and train neural networks. Um so the re the reason TensorFlow is called

[7:15:18] TensorFlow is primarily because uh the fact that it manages this underlying graph of tensor. So all the underlying graph of tensor. So all the data that flows through this graph um is

[7:15:31] actually in uh this tensor object which is very similar to kind of a numpy array or kind of like a pandas dataf frame. very simple uh very similar kind of

[7:15:44] idea. So, but in reality we can think of a tensor as kind of a kind of a a a a tensor as kind of a kind of a a a matrix um like kind of data object that

[7:15:56] uh contains kind of multi-dimensional arrays inside of the tensor. So um you know we've been dealing with data that's in a matrix format in terms of a data frame. So this was our data frame. um

[7:16:10] the vector was kind of our numpy array um kind of structure. And now in TensorFlow we're going to be working with tensors which are going to be these kind of like matrices of matrices essentially. And uh so TensorFlow

[7:16:27] essentially. And uh so TensorFlow manipulates data uh that is uh converted into tensor uh into tensors under the hood of TensorFlow whenever we pass data into our into the network. Um so it's just useful to know that is that the

[7:16:42] data that TensorFlow is manipulating is actually tensors and not really data frames, not really numpy arrays. But that's not to say we can't start with this kind of data and use it with TensorFlow. Of course we can and

[7:16:55] TensorFlow will manage converting data frames or converting numpy arrays into tensors under the hood. That's something that they do. But it manages kind of the computations in this tensor kind of data structure in the same way that you know

[7:17:11] the the models that we've built to this point mainly manipulate things like data frames, right? They they manage and manipulate data frames. TensorFlow manipulates tensors. Um but luckily you know it will convert things like dataf

[7:17:25] frames into a tensor as needed. So we don't really need to worry too much about that. Just pointing it out. That's where it kind of gets its name is that structure that tensor structure. Okay. So mainly what we're going to be using

[7:17:39] So mainly what we're going to be using TensorFlow for is of course for uh building and training neural nets. But TensorFlow as a library has a lot of TensorFlow as a library has a lot of other features to it. um it has these uh

[7:17:52] it has things on top of it that make life so much easier when it comes to what we're talking about here with these highle um APIs which are which would be highle um APIs which are which would be something like uh caris

[7:18:13] to manipulate like build and train neural networks it'll be really easy to do with a library like caris uh built on top of TensorFlow. TensorFlow also has a bunch of models that are built into it that are already trained. That'll be

[7:18:27] really useful for things like computer vision um where we want to pull things off the shelf and maybe fine-tune it um or do what's called transfer learning to it to to make it better on our own data. So that's really cool. It has a bunch of

[7:18:40] pre-trained models already built in. Um it can take advantage of GPUs really effectively. So, TensorFlow manages GPU uh compute really, really nicely. It interfaces with GPUs really easily, which is a great feature for it

[7:18:56] because uh we're going to need that especially to train um especially to train computer vision uh models later on down the line like when we get into object detection things like that. GPU is going to be really important. Um I

[7:19:10] already mentioned TensorBoard. That's a really important visualization uh kind really important visualization uh kind of library that TensorFlow provides to be able to to plot a graph like this in real time where we can kind of see the

[7:19:22] loss over the course of the iterations and see that kind of updating in real time as iterations are finishing kind of see how that loss is progressing and and that can inform us if we need to stop if we should um let it keep going all those

[7:19:38] visualize with the help of TensorBoard. So we'll see that later on. As I said So we'll see that later on. As I said earlier, um TensorFlow also has features that help for model deployment. Now, we won't really use that too much. Um but

[7:19:53] you can build a neural network and kind of easily um deploy it behind an endpoint. We won't really need to do that. Um but uh it does facilitate that

[7:20:06] in in the library. So if we needed to, we could easily do that. Okay. So I hope it makes sense why TensorFlow was such a big deal, especially after going through our examples of like forward and backward propagation. When you're

[7:20:21] backward propagation. When you're building a neural network, there's so much computation to keep track of. And we we saw that, right? We saw that kind of firsthand is you know forward propagation you're multiplying and doing

[7:20:34] this uh weighted sum through an activation function and you're doing that and passing that data forward throughout the network and that's a lot of computation especially if it's a deep network but man that back propagation

[7:20:48] network but man that back propagation was so so so tedious right computing the gradients that is challenging um and then keeping track of that and multip multiplying those together with the right terms and propagating that

[7:21:03] backward throughout the whole network is a very tedious and intensive process but libraries like TensorFlow and PyTorch manage that for us which is a huge deal

[7:21:15] manage that for us which is a huge deal and that's why they were created to really manage that computation and help us train and build neural networks. So that's why we need a library like TensorFlow is to ultimately

[7:21:29] manage the computation that is inherent in a neural network. There's a lot it's much more complex than any kind of machine learning model, right? There's a lot more going on with it. And so it necessitates these kind of libraries

[7:21:43] that are specialized in in neural nets beyond scikitlearn. Scikitlearn was great. We've used scikitlearn for you know so many machine learning models but we really need to go beyond it to the specialized libraries for neural nets

[7:21:59] because there's so much going on with those. Okay. So, as I mentioned before, um, under the hood of TensorFlow, and it's not like we necessarily need to know this, but just showing you kind of how it works, is TensorFlow will manage

[7:22:14] the network and break it down into really manageable chunks of computation really manageable chunks of computation by um kind of organizing, organizing that computation into really really manageable pieces. So for

[7:22:30] [snorts] example, doing a weighted sum and passing that through an activation kind of gets broken down into a data flow graph that looks like this. So you can even see this here like we do a a a matrix multiplication between the

[7:22:44] weights and the inputs and that forms our weighted sum. That's just a dotproduct, right? Weighted sum. Uh and then we add in a bias. So we add

[7:22:56] in a bias which is what this does. So it manages adding a bias term here and then it passes that into a relu function which is doing our activation on this weighted sum. Right? So it everything that we've been

[7:23:12] doing this this weighted sum through an activation efficiently uh to produce an output. And of course the uh the the deeper the

[7:23:28] network the more sophisticated this graph looks. But this is just a simple this would be like the perceptron righted some through an activation. It looks just like this a and on and on and on. So a more complex uh neural network

[7:23:42] is going to be represented by a much bigger graph here. But TensorFlow manages that under the hood and it does that on purpose so that it can manage all aspects of the neural net. It can manage training. It can manage

[7:23:55] prediction by doing forward propagation like this. It can manage back propagation by doing all these gradients really effectively um in of the graph. So by putting it in this graph structure, it's able to

[7:24:10] manage our computation so so effectively. Um that's the beauty of TensorFlow is this data flow graph of tensors. um and it manages it really really efficiently. Okay, so basically

[7:24:26] everything I said is uh TensorFlow manages the computation through this manages the computation through this graph. Um and it can also leverage graph. Um and it can also leverage parallelism by using GPUs. So it can uh

[7:24:39] parallelism by using GPUs. So it can uh basically optimize this graph execution basically optimize this graph execution um as it needs to parallelize things as offload a lot of these calculations in the graph to a GPU and that's something

[7:24:53] the graph to a GPU and that's something we will uh work with later on. Um so it's in that way very effective at managing again managing the computation. So some of the benefits as I said using GPUs it can uh parallelize things um it

[7:25:08] can be built in Python this graph can actually be saved and reloaded really effectively. So the the idea of putting things in a graph like this makes it things in a graph like this makes it really um persist persistent like we can

[7:25:22] persist this graph uh architecture very easily and then reload it as we need to. So in that sense we can always save our architectures and save any of our weights that are there like this weight matrix um we can save and so like

[7:25:37] loading and saving models actually becomes really really easy in TensorFlow and same thing for PyTorch. Everything that I'm saying here for TensorFlow is honestly applicable to PyTorch as well. I know we're going to cover that in

[7:25:49] I know we're going to cover that in lesson six, but um very similar where PyTorch manages everything as kind of a data graph. And and remember PyTorch was made by Facebook. So they kind of made these libraries independent of each

[7:26:03] other to both try to tackle the neural network problem and then they open sourced them uh to make it available for the community later down the road. But um you know Google and Facebook were both working on neural networks quite a

[7:26:17] bit. So they had their own libraries to manage those and they're both prominent today right TensorFlow and PyTorch. So when it comes to working with TensorFlow when it comes to working with TensorFlow um there's kind of two approaches.

[7:26:31] um there's kind of two approaches. One is to work at that graph level um which is kind of the low level of TensorFlow. Um so working at the graph level is sometimes known as TensorFlow core. It's just the core of TensorFlow.

[7:26:46] We won't do that. It's very complicated to do that. Um it it gives you ultimate control over modeling though because you're controlling that data tensor you're controlling that data tensor graph um directly. So you get a lot of

[7:26:59] control and fine grain control over building your models. We won't ever need building your models. We won't ever need to do that. um we'll mostly work at the caris level which is kind of an abstraction layer on top of the graph.

[7:27:11] So, caris is going to be a library that sits on top of tensorflow almost in the way that like a seaborn sits on top of mplot lib

[7:27:23] or a panda sits on top of numpy very similar way. Caris kind of is an similar way. Caris kind of is an interface to the low-level graphs and it makes it much much easier to manage uh neural networks and training um because

[7:27:40] we don't need to go into the graph level details [snorts] um we can just stick at kind of a high level of building the models. So most everyone actually building neural networks will utilize caris. If you're

[7:27:54] networks will utilize caris. If you're doing deep research and writing a paper performance and coming up with new architectures, you may work at the graph level. Um we're not going to do that in this program because nobody really does

[7:28:09] that for the purpose of building a model. You you would generally work at the caris level. Uh which is what we're going to do. Okay. So, makes it easier and that's honestly what is practical is to work at that level because um you

[7:28:24] a lot in terms of building really effective models. It's just we're not going to be coming up with brand. So, if you were to come up with a brand new layer like an attention layer for transformers, you would be doing at that

[7:28:38] at the graph level, the very low low level. We're not assuming we're going to do that. We're just going to use what's already uh basically in in our toolbox inside of Caris and build models off of that which is plenty of complexity for

[7:28:52] that which is plenty of complexity for us. Okay. So just to uh oh sorry just to reiterate you know some of the the uh features of TensorFlow um is that we are going to be it's it's open source so it's easy to install even

[7:29:08] really easy to install. You could just pip install it just like you would any pip install it just like you would any other Python library. It is free. Um and it supports uh building basically doing GPU training um which is uh really

[7:29:25] really nice. So TensorFlow will interact with will interact with GPUs really really effectively which is nice. Okay. So here's some of the um advantages of working with TensorFlow. We've already talked about a lot of this open source.

[7:29:38] uh it does GPU computation really easily, very flexible library in terms of the uh ability to build out different kinds of neural networks. It actually contains a lot of different layers that we may need. We're going to see that

[7:29:52] when we start building our own networks. The fact that it also has highle uh The fact that it has a caris, we don't need to go into the graph level makes it need to go into the graph level makes it super approachable for us uh to get

[7:30:07] would build and train neural nets. It's so much more approachable than having to get into the graph level details which is very complex. Uh do we have to cloud? No, you can work in the cloud. Uh so collab has tensorflow built into it

[7:30:22] because it you know um tensorflow came from Google. So, uh, of course, Google Collab is going to have TensorFlow built in. You don't need to install it. Um, so every Collab notebook can work with TensorFlow. In fact, it's actually

[7:30:36] really nice to work in Collab for this stuff because I' I think I've said before, Collab has GPUs for free. So, you can connect to a GPU and kind of you can connect to a GPU and kind of train your models on GPU really easily

[7:30:49] in the Collab environment. So I definitely encourage especially for deep learning to try to use Collab if you can. I think so TensorFlow is also installed in the SimplyLearn lab environment. So if you want to use that

[7:31:02] environment, it also has a GPU connected to it for free. Um so you can feel free to use that and and it has TensorFlow set up in it. You would just have to install if you want to run locally. You probably would have to install because

[7:31:16] it's likely you don't have the TensorFlow package. Um, and I can share TensorFlow package. Um, and I can share some resources later on uh for utilizing the GPU uh locally, especially if you're like on a M1, M2, M3, M4 Mac, you can

[7:31:30] actually use the GPUs uh in the in the uh M1, M2, M3, M4 chips. Um, you can actually use GPUs locally on those, which is really nice. Or or if you have like an Nvidia GPU, you can use that too either way. But I do recommend Collab

[7:31:47] strongly. um you know because of it uh free GPU ability and it and it all just works pretty seamlessly in Collab. Okay, so what's nice about uh TensorFlow as

[7:31:59] well is obviously everything's going to be in Python for us to be able to do to handle all of our neural network building and training. um that'll all be in Python. But actually there's a lot under the hood, you know, not that we

[7:32:13] need to know this, but under the hood of TensorFlow is a lot of like efficient C implementations that make work that make the graph data flow uh as efficient as

[7:32:25] possible. So it's Python, but it's powered by C honestly a lot under the hood. Um which is which is a lot of libraries are like that. Numpy is very like that. Um, it's basically an interface to low-level C code that's

[7:32:40] manipulating all the the numpy arrays. So, TensorFlow is kind of no different. Um, has very efficient uh C implementation. Um, it also has the parallel uh computation. It has a it has

[7:32:54] parallel computation capabilities. This is something we're going to see uh that we can easily offload our computations to a GPU to get to to have as much as it to a GPU to get to to have as much as it of it parallelized as we possibly can to

[7:33:08] speed up our computations. Um so what we're going to see is TensorFlow makes it and PyTorch PyTorch for that matter too they both make it really easy to do that. So we'll see that later on. TensorFlow can run on many different

[7:33:24] environments including mobile and it's really important mobile to call out mobile on this mainly because a lot of sophisticated AI

[7:33:36] because a lot of sophisticated AI programs that run on phones like the Apple intelligence for example are often powered by neural networks that have to powered by neural networks that have to run on a mobile device. Um so that's so

[7:33:49] t TensorFlow would be responsible for running that model and so it has to be compatible with uh kind of uh edge devices like like an iOS or Android or

[7:34:01] uh even like a Raspberry Pi which is kind of a small uh computation platform. So, TensorFlow works across many different platforms, which is great neural networks on many different platforms, including those kind of

[7:34:16] mobile edge devices. Now, because it's open source, it has a really vibrant community. It's one of the most popular open source projects on GitHub. Um, it still has a very dedicated team at Google that that's dedicated to kind of

[7:34:32] source. there's developers there that are kind of, you know, focused on maintaining it and making sure it's in a good state. Um, but it definitely has a things that comes out of that community is really, really good examples and

[7:34:46] documentation. So, I'll show us that probably later on today. The the TensorFlow documentation is really, really good. So, we'll utilize that in certain areas when we need to. Okay. So what we're going to be interested in and

[7:35:00] actually get into this soon and then start building as well in our code. So that's coming up shortly uh is using highle abstractions like caris. So the the graphs are great but we don't want to manage things at a graph level that's

[7:35:16] to manage things at a graph level that's very very uh complex as I've alluded to. So, we want to be able to um basically quickly prototype models, quickly develop them, and quickly train them. And we'll be able to do that using

[7:35:30] something like Caris because it'll make uh building a neural net and training it really only take a handful of lines of code, which is really amazing. Um it's it's actually remarkable how these libraries like TensorFlow and PyTorch

[7:35:46] how easy they make building and training a neural net. They make it really easy to do which is nice and that's we've seen that before with you know really easy by basically making everything into a ffit and predict right

[7:36:03] um so so caris will be kind of the same thing and pietorch kind of the same thing and pietorch kind of the same thing making our life really easy uh in terms of building and training neural nets so many different uh applications

[7:36:16] that use tensorflow models that are out there today So image recognition, videos, NLP, basically anything that uses neural nets, um transformers, of course, not listed on here, but anything that's doing it, you know, anything with

[7:36:31] neural nets. Um it often there is a TensorFlow model for it. Doesn't mean it's exclusively in TensorFlow. There may be a PyTorch version of it as well,

[7:36:44] but most of the time there's there's both, you know, a TensorFlow version and a PyTorch version for many different models that are out there. Um, they're generally developed in one of those two frameworks, if not both. So many

[7:36:57] companies leverage TensorFlow because they are using neural nets to do they are using neural nets to do something. Um so like object detection uh Uber you know uh and a lot of self-driving uh is powered by neural

[7:37:11] nets that is that TensorFlow uh helps provide and many companies are using TensorFlow for different kinds of uh neural net applications of course so no surprise here to see a lot of big companies on this list Google not

[7:37:27] shocking they created it of course they're on the list so some application areas I I mean we've talked about these before like medical images uh diagnosing before like medical images uh diagnosing disease um they so Google for example

[7:37:40] created uh this application called germist using tensorflow um basically to make diagnosis on skin conditions which is like pretty interesting using just is like pretty interesting using just pictures um so so many like neural net

[7:37:54] I mean so many are developed in pietorch as well those two libraries just dominate the market in terms of building and training neural nets. So, uh the the

[7:38:07] and training neural nets. So, uh the the Twitter or now X their uh ranking of tweets uh to do like recommendation is uh all powered by TensorFlow. Google uh all powered by TensorFlow. Google using uh uses TensorFlow to do its

[7:38:21] using uh uses TensorFlow to do its search rankings. Photo sharing VS Code um uses TensorFlow to suggest filters. So so many TensorFlow applications that are out there uh that are in use today. I mean not not to say there's not so

[7:38:36] many PyTorch examples. There are definitely those as well. But a lot of them, you know, a lot of people chose TensorFlow, a lot of people chose PyTorch. They're both pretty represented in the in the community. Um so we can

[7:38:51] use it for text applications. Um and you know a lot of uh even up to LLMs like transformer-based models are built in TensorFlow um or PyTorch or

[7:39:04] both as I said both very very popular frameworks. So NLP applications tensorflow has been used for uh e-commerce use tensorflows for recommendations, you know, we studied recommendations and saw a lot of

[7:39:20] techniques to do that like um collaborative filtering and matrix factorization. Those are still used in some places, but a lot of places have graduated to doing neural networkbased techniques. And uh

[7:39:34] nets, chances are you're using TensorFlow or PyTorch um to do that. And so so many companies have uh adopted TensorFlow to do recommendation

[7:39:46] uh because they're using some type of neural net to do that. Pretty interesting. Okay. So what I wanted to interesting. Okay. So what I wanted to do is um show you one short demo on

[7:39:58] kind of see what the tensor data structure looks like. Um so I'm going to do that. I'm going to do this demo to start with. So um we're going to start with the 5.0 02 demo from lesson 5,

[7:40:12] introduction to tensors as a data type. And what you're going to see is tensors are very much like numpy arrays, very much like data frames. They're just a basic data structure that TensorFlow works with. Um, now these these two

[7:40:27] demos I'm going to hold off on because they get into building a neural net. What I want us to see first is to jump ahead to the caris notes here in lesson five. So we're going to cover caris first and then once we've covered

[7:40:44] caris in a few slides, work our way back to these two demos. So the plan for today is I'm going to cover this demo first. We're going to jump ahead to the caris notes to get some information on how we can use caris to build neural

[7:40:59] nets and then get our hands on these two demos so we can see it in action. So these two demos are going to have kind of end toend examples on building and training neural nets. Obviously we haven't learned that yet. So I want to

[7:41:11] jump ahead to car so we can see some of that and then work our way back to these two demos to to actually practice that. Um so so that'll be kind of the plan. Um, so with that being said, let me let me uh jump over to the 5.02

[7:41:28] um notebook um just so we can see some basic examples with just tensors as kind of a data structure. Okay, do you guys have this one? 5.02. This is the uh first demo in lesson five.

[7:41:43] first demo in lesson five. Okay, great. you shouldn't need to run this. In fact, you can see it already says everything is satisfied if you do run this. This is just like for your local if you're if

[7:41:57] machine, you probably need to install these because you probably don't have these because you probably don't have them. So, this would be uh for local them. So, this would be uh for local machine. Um if you don't have these

[7:42:12] um you'd want to install them. They're already here in Collab. So, it just tells us, hey, we already have all these. Um so nothing really happens.

[7:42:28] All right. So then we have some setup here. Now these things are optimizations that that are uh basically flags for TensorFlow to know if it should be using

[7:42:40] GPU or if it should be using CPU and if it should be making optimizations based on the GPU. So, what we can do is um basically disable GPU for this demo because we're not going to need GPU to

[7:42:54] do anything with tensors yet. Um we're only going to need GPU for uh later on when we're actually training uh more sophisticated models that that are going sophisticated models that that are going to require it. So, this is basically

[7:43:08] setting um the we're going to turn off GPU visibility here. Um so, we don't use a GPU in any in any way because we don't need to right now. It's kind of a waste.

[7:43:20] Uh it wouldn't be useful to us. And we're going to set the uh this is just some logging output from TensorFlow uh in case there's any warnings or in case there's any warnings or anything. Um so, we'll just run that.

[7:43:34] anything. Um so, we'll just run that. Okay. So what I wanted to show you is uh first of all TensorFlow as an import is kind of like many other packages in data kind of like many other packages in data science and machine learning. It has an

[7:43:47] alias that is kind of an industry standard alias which is TF. So um this standard alias which is TF. So um this is a industry standard alias is TF for

[7:43:59] um TensorFlow right? So we can see is we import right? So we can see is we import tensorflow as TF um which is kind of the industry standard to alias it. So you know just like we do numpy as MP, pandas

[7:44:13] as pd um tensorflow is typically imported as TF. That's very industry standard. So if somebody was reading your code and came across and saw TF, they would know what that means. They know it's TensorFlow

[7:44:27] that means. They know it's TensorFlow that we're using. Okay. that we're using. Okay. So in TensorFlow um as I said the the manipulated under the hood that kind of holds our data um for training and doing

[7:44:42] prediction with a neural net is known as a tensor. Um and you can create different dimensional tensors uh inside of TensorFlows or inside of TensorFlow I should say. Um the very ba the most

[7:44:59] basic tensor you can possibly have basically is dimension zero also known basically is dimension zero also known as rank zero. So rank is equivalent to kind of the shape or the dimension that we had uh from like a numpy array or a

[7:45:15] we had uh from like a numpy array or a pandas dataf frame. So a rank zero is just a constant. It's it basically has um no dimensions to it. it's just a um no dimensions to it. it's just a single value. So um in TensorFlow you

[7:45:28] create a single value with TF.stant. So this creates just a single scalar value. It basically has no array dimensions um no matrix dimensions just a sing it's like a zerodimensional numpy array um

[7:45:43] like a zerodimensional numpy array um equivalent. So what you can see is we create the tensor here and then we print it out. What you can see when we print it out is look at the data structure that it prints. It prints a TF tensor.

[7:45:59] That's mainly what um we're seeing here is is this is the kind of data that TensorFlow is going to work with is a TF.tensor structure. So the the TF I should call that out here. The TF.tensor

[7:46:15] that out here. The TF.tensor structure is um equivalent to a mparray structure or a pd. dataf frame

[7:46:29] uh structure. It's it's equivalent to those but it's just in TensorFlow. This is the data the primary data structure that's used is a is a tensor TF tensor. This has no shape to it because it's zero dimensional. So it's just a single

[7:46:45] constant value of four. It's a scaler and you can see what its data type is is an is an int32 which is just the default integer uh type for for integer data.

[7:46:59] Okay. So we could make a onedimensional tensor which is basically just an array with basically a vector that has a few values in it. Um so a onedimensional is

[7:47:11] values in it. Um so a onedimensional is still a um a tf.stant constant, but you still a um a tf.stant constant, but you can see and now we have a a list or you could put a list. You could also put a numpy array there. Um, you can actually

[7:47:25] build a tensor out of either kind of data. Um, and so this is just a single dimensional vector. And what I want you to see is look at the tensor now. So the

[7:47:37] to see is look at the tensor now. So the tensor now has this array as its data. tensor now has this array as its data. And it now has a shape of three uh comma nothing which is the signal that it only has one dimension and that dimension has

[7:47:51] three elements in its shape. Right? So it's a shape of three nothing cuz it's just a single dimension. So that's a rank that's what we would call a rank dimension. But again the underlying data is a TF.tensor.

[7:48:05] This is what is manipulated under the hood of any neural net is a tensor structure. So just showing you this so you can kind of see what actually is being manip manipulated. Um when we build a neural net it's really this

[7:48:22] build a neural net it's really this tf.tensors. tensor which is a matrix. So this is data that has rows and columns. It's

[7:48:35] two-dimensional. So here we have a 3x two. We have three rows and two columns of data. And the other thing we can do when we build a a tensor. So we use when we build a a tensor. So we use tf.constant to build the tensor. Um and

[7:48:51] so we put in this two-dimensional array. We also specify that the data type should be float 16. So that's really interesting. The original data is just integers, but we actually have the flexibility to say, "Hey, I want you to

[7:49:06] flexibility to say, "Hey, I want you to put all of this in float 16 data type." So you can see when we create this tensor, it has this data and it's um tensor, it has this data and it's um kind of casted over to a float 16 data

[7:49:18] kind of casted over to a float 16 data type. Okay. So that is a two-dimensional and we could create even a threedimensional just like we've done before with with uh you know threedimensional numpy arrays. So we

[7:49:31] threedimensional numpy arrays. So we have um you know this is now this is basically uh every entry in this tensor is a matrix itself. So we have three matrices. Each matrix has two rows and

[7:49:45] five columns. That's what these dimensions really represent. So we have dimensions really represent. So we have one matrix here, one matrix here and one matrix here. So we have three of them. Each of them are 2x5

[7:49:58] Each of them are 2x5 and so we get a shape of 3x 2x 5. So um three dimensions are actually very popular in TensorFlow because um usually

[7:50:11] the data will be in either three or actually a lot of times four dimensions because um this first dimension may represent like a batch like we may have

[7:50:23] represent like a batch like we may have um 100 images and each image is this resolution which is maybe like 28x 28 resolution um like as a matrix. So three dimensions is actually very very common for a

[7:50:36] is actually very very common for a tensor and and actually four dimensions tensor and and actually four dimensions is also very very common um because the first dimension is still the batch but every image may be a 28x 28 resolution

[7:50:50] and then have like three color channels like RGB, red, green, blue. Um so four dimensions is actually pretty common. 100 images each one is 28x 28 with three

[7:51:04] colors. This is actually so four dimensions is actually very very uh common in TensorFlow when we're working with uh data and building like image based networks that have color images. Um

[7:51:19] that's actually uh pretty common to see four-dimensional data. you can actually go back and forth between tensors and numpy arrays. So,

[7:51:35] between tensors and numpy arrays. So, all you have to do is um just call numpy on it and this will convert any tensor over to a numpy array. So, this is a uh numpy array and the the tensorflow float 16 just converts over to a numpy float

[7:51:52] 16 just converts over to a numpy float 16 type um very seamlessly. So it it just retains that kind of type just in numpy. Okay. So if you ever want to convert for whatever reason, maybe you want to manipulate that tensor in numpy

[7:52:05] directly, um you could just call numpy on it and it will turn that tensor into a numpy array with the same shape. So now we have that two-dimensional array now we have that two-dimensional array there.

[7:52:22] course going to be operations between tensors which needs to happen in order tensors which needs to happen in order to do matrix uh manipulations and to do uh those kind of like weighted sums through activations and all the kind of

[7:52:35] computations that we need to do. There has to be uh mathematical operations between tensors. they need to support that in order to do any kind of meaningful computation with the neural network. Right? So what you can see in

[7:52:50] network. Right? So what you can see in this example is just some examples of uh adding multiplying and doing matrix multiplication between two tensors. So multiplication between two tensors. So we have two tensors which are 2x two uh

[7:53:03] we have two tensors which are 2x two uh rank two tensors and we can add them together which will do an elementwise addition. We could multiply them together which will do an element-wise multiplication or we can do a

[7:53:15] traditional matrix multiplication which is that remember matrix multiplication is that row by column dotproduct constructing a new matrix out of that. Um and and we can do all those

[7:53:28] all of those operations. Um so this makes sense. This like tensorflow needs to support these kind of operations to do any meaningful kind of manipulations right of underlying data. Again will we need to do this

[7:53:45] like no we won't really need to do this because this will be handled under the hood by the highle library like caris. they will in TensorFlow will manage doing all these computations but when they do those computations this is what

[7:54:00] they're doing under the hood multiplication matrix multiplication addition um passing it through an activation all those kind of operations are are managed by TensorFlow

[7:54:14] which is really nice. It has other uh operations as well. Things like finding the max which makes sense because sometimes we need to find the max in order to uh in order to calculate like the maximum probability

[7:54:30] or find out which class something should be which is the arg max. So this is be which is the arg max. So this is finding the index where the maximum is. That's an important operation for figuring out like which which uh entry

[7:54:43] in this array has the highest probability. Um that's the class prediction is the argaxs and then what I wanted to show you too is the uh TensorFlow has builtin activation functions. So you can see one

[7:54:59] of them here and we're going to get into a bunch of them when we start building our neural nets is like the softmax activation. So you can see um we take activation. So you can see um we take this uh we take this um tensor C which

[7:55:13] is a two-dimensional tensor and we actually compute the soft max of it which will normalize it right into probabilities that uh total up to one uh

[7:55:25] across the rows. So because it's two dimensions um the softmax will make sure dimensions um the softmax will make sure that like this adds up to one and then this adds up to one. So every row adds up to one. So this is just an example of

[7:55:41] applying an activation like a softmax. Um notice where this come from comes from is the NN module which is the neural net module. So uh that's from its built-in neural net module. It has a bunch of stuff like

[7:55:56] activations um those kind of things. We'll talk more um those kind of things. We'll talk more about that as we get into caris.

[7:56:09] to run the notebook? All these steps have ran for you guys. Hopefully, no issues. We're not doing anything fancy yet. Just manipulating tensors, which are hopefully you guys can see are mostly acting like a data frame or a

[7:56:23] numpy array. Mostly acting in the same way. Um and then finally just like numpy arrays um you know TensorFlow does

[7:56:37] about broadcasting before when we did numpy arrays. Essentially if you take um a vector or in this case I should say a tensor and multiply it by uh a constant

[7:56:51] um then that will just apply that constant then that will just apply that constant to every entry in that vector. Um which makes sense. And then um otherwise like if you multiply uh this times this um

[7:57:06] it'll just do an elementwise um if you do you know it if you try to um if you do you know it if you try to multiply um different shapes together it will attempt to broadcast to make the shapes fit and it will give you an error

[7:57:21] if it cannot do that. Um, so most of the time it's going to try to do elementwise multiplication and then attempt to do some broadcasting if the shapes don't line up, which we saw exactly, you know, the same exact thing happen with numpy

[7:57:35] arrays. So that's that's not entirely shocking that that TensorFlow would shocking that that TensorFlow would support broadcasting in some way. So, and the so a couple more operations you can actually reshape tensors which

[7:57:48] you can actually reshape tensors which sometimes is useful uh that for for um that'll happen underneath the hood of TensorFlow anyways like it needs to do a TensorFlow anyways like it needs to do a reshape to uh um manipulate properly. So

[7:58:03] we can take this tensor and kind of reshape it into a 1x3 um which if you print it out would now uh you know would just kind of transpose uh you know would just kind of transpose it um essentially from 3x one into 1x3.

[7:58:19] Um, so you can do so what I hope you're seeing with this and part of the reason to go through this is I hope what you're seeing is a lot of the same stuff that we had for dataf frames and numpy arrays

[7:58:32] is really carrying over to tensors. In fact, it's kind of inspired by that. When Google made TensorFlow um they were kind of inspired by what was already out there in terms of numpy and pandas. So it makes sense that

[7:58:48] they're going to carry over a lot of the same operations like reshape and and uh having things that look like numpy arrays and pandas data frames. That makes sense. Um

[7:59:03] finally there's uh technically in TensorFlow they have different types of tensors like you can have what's called a ragged tensor which is um basically having allowing for different size dimensions. Um so you can have instead

[7:59:19] of requiring everything to have the same number of uh columns essentially like this first and this first row has four columns but this next row only has two. this next row only has three and then this last row only has one. So we

[7:59:33] basically pack that in as kind of like a list of lists um in this what's called a ragged tensor. We won't really ever have a use for using ragged tensors, but just to say that TensorFlow supports having

[7:59:46] different size arrays and inside of a tensor object. Um you can even have tensor object. Um you can even have strings um inside of a tensor. So you can see here um this string gets stored here. Um, and you can see its data type

[8:00:00] is string. So, it is possible to manipulate strings. That's not very common because most of the time when TensorFlow is manipulating strings, uh, they'll they'll mostly be like in like an LLM, they'll be converted over into

[8:00:15] vectors before they're ever manipulated. So, very rare you would actually of TensorFlow, but it does support. You can actually put in string data into a tensor and it can hold that data. It just won't be able to do much with it

[8:00:30] until it's converted over to a vector. Uh which is what we see with kind of language models. They tend to convert strings into vectors before doing any manipulation with it. Um we'll learn about that later.

[8:00:45] about that later. All right. So let's talk about uh caras. Um, so just a reminder of what Caris is, it's basically a an interface built on

[8:00:58] top of TensorFlow to help us build and train neural network models without having to go into the weeds of the graph interface of core TensorFlow. So it's

[8:01:10] very very nice in that way. It's really easy to use um really fast to build models and train them uh in terms of getting up and running with it. really user friendly. That's what it was built for is to be really user friendly. And

[8:01:24] so it's kind of everyone's preferred way of interacting with TensorFlow. Um, if you can because it's so nice to use as we're going to see, you know, as I said earlier, unless we're doing very very sophisticated research, there's no

[8:01:40] reason we shouldn't be using caris and uh people wouldn't people generally prefer using caris. It's so nice to use for building and and working with neural nets. Um, so of course it's in Python and it uh it now caris also does the

[8:01:58] same thing as TensorFlow in the sense that it allows us to use GPUs relatively that it allows us to use GPUs relatively easy e easier uh or easily I should say. easy e easier uh or easily I should say. Um, and it's uh really really easy to

[8:02:11] basically prototype and build your your neural nets and get up and running really quickly that way. So I think one of the ways to think about KAS is I I use the analogy earlier of kind of like how we have pandas sitting on top of

[8:02:27] how we have pandas sitting on top of numpy. The underlying data in a dataf frame is technically stored in a numpy array. Um, but we don't work with it in a numpy array style. We work with it in a dataf frame. Because when it's in a

[8:02:41] dataf frame in pandas, there's so much we can do with that data frame from a pandas perspective, right? We can manipulate it. We can um aggregate it. We can join it to other data frames. So many things we can do with a dataf frame

[8:02:55] that we cannot do with just a basic numpy array. Same thing with caris. There's so much we can do with caris to help us build neural nets and train it um that it it's you know sits on top of tensorflow and the underlying tensor

[8:03:10] graphs and it just abstracts that all away for us and focuses on the core things we care about like layers loss functions um setting up the training and all the hyperparameters around training. So it

[8:03:26] working with just the underlying TensorFlow which can be challenging and and take more time to ramp up to for sure. Basically as I said t uh caris acts as an interface for it. Um and we only need

[8:03:40] running building any neural nets which is really awesome. So uh caris actually is really awesome. So uh caris actually has extensions to other deep learning libraries. Um so not that we need to know that but it caris has been extended

[8:03:55] to be an interface to many other um backends that are neural network backends that are neural network libraries. So many of them are are way TensorFlow. So of course we're going to focus our efforts on working with caris

[8:04:10] as an interface to tensorflow. But these guys here are other deep learning guys here are other deep learning libraries that Caris has support for as libraries that Caris has support for as well, but they're not nearly as popular

[8:04:25] well, but they're not nearly as popular as PyTorch or uh or or TensorFlow. Um, unfortunately there's no caris version for PyTorch, but that's okay because PyTorch is kind of its own has its own um abstractions on top of the underlying

[8:04:41] tensor graph structure of PyTorch. So we'll learn about that in lesson six. So just to say caris has grown really popular and has been extended to other to other deep learning libraries, but the other ones on this list are not

[8:04:56] TensorFlow kind of dominates the industry along with PyTorch. PyTorch and TensorFlow blow away all these others in terms of their usage. Um so just uh continuing the theme here of being really nice uh timesaving. We don't have

[8:05:11] really nice uh timesaving. We don't have to go into uh learning the underlying uh core TensorFlow which I think from like a learner perspective from your guys's perspective is so nice. You don't have to spend a lot of time getting ramped up

[8:05:24] to spend a lot of time getting ramped up on the the nuts and bolts of core TensorFlow which is really challenging. It's not an easy library to understand. Um but Caris is caris is really easy to work with. So it makes it really user

[8:05:37] friendly to get up and running. We just talked about this. Many backends are talked about this. Many backends are supported from uh from TensorFlow and or from Caris. And one of the reasons is like as I said before TensorFlow was

[8:05:52] open- sourced a lot of people were developing their own neural net libraries. So these have been around like the MXNet CNTK which is from Microsoft. These are neural net frameworks that a lot of people were

[8:06:06] building because they needed them to be able to manipulate um or to build their own neural nets. But TensorFlow emerged and PyTorch emerged as the dominant libraries. Um, Caris is the main way that we interact with TensorFlow, which

[8:06:22] is why we care about Caris. Um, but Caris has extensions to be able to work with other neural net libraries. It's just those others are a fraction of what

[8:06:34] just those others are a fraction of what we see um, PyTorch and TensorFlow uh, being used for these days. Okay, so no need to keep selling you on it. Cares is really important. uh and of course that's what we're going to uh be using

[8:06:48] here initially to build out some of our to build out our models and train them. the other thing is it has really good documentation source community. Caris is open source just like TensorFlow is. So, uh has

[8:07:04] just like TensorFlow is. So, uh has really good um support for uh people uh maintaining it and building out documentation and things um still to this day. Okay, so we've talked about all of this. We've talked about how

[8:07:18] userfriendly and how easy it is. Quality documentation we just talked about. In documentation coming up before we uh finish today. So you can see kind of what that documentation looks like in terms of um a way to see all of the

[8:07:34] all the activations that are available, the way to see all the loss functions that are available from Caris. Um so I'll show us that multiple backends we've talked about, but we don't really

[8:07:47] be using it with TensorFlow primarily. One thing that's cool is uh just like underlying TensorFlow, Caris has easy ways to access pre-trained models which will be important in like computer vision or even in NLP to grab models off

[8:08:03] the shelf so we can kind of use them for our use cases. Um so that'll be a nice feature. Caris allows us to work with GPUs. It can even work with multiple GPUs if you have them. Um so that's also really nice and it makes sense like

[8:08:18] underlying tensorflow supports that so of course caris is going to make it easy to do that too. Okay so one of the let's start getting into some of the elements of caris that we are going to use to construct our neural networks. So the

[8:08:33] construct our neural networks. So the first place to start is going to be the first place to start is going to be the uh layer um class. Now there's many different versions of layers that are inside of caris and I'll show us a

[8:08:46] inside of caris and I'll show us a documentation of that later. But um a layer is a fundamental building block of any model. And so when we build a layer,

[8:08:58] one of the things we need to decide when we build a layer is how many neurons do we build a layer is how many neurons do we want in that layer. Um and so that's going to be an important input. The other input that we'll typically need

[8:09:11] when we construct a layer is the activation. So what activation do we to apply relu? Do you want to apply softmax? Do you want to apply sigmoid?

[8:09:23] Uh hyperbolic tangent? You know, what activation do we want to have inside of all these neurons across this layer? So that's going to be important. Um, and

[8:09:35] what's interesting is we will be able to connect layers together inside of a neural net, we're going to be building out basically declaring layers and then

[8:09:48] connecting them together inside of this network structure. So we're going to see that in caris. But the fundamental building block is going to come from this uh module right here. The tensorflow

[8:10:02] layers. And then there's going to be many different kinds of layers that are possible. This is just a generic layer, but there's going to be many different kinds of layers that are available for us to use. For example, there's going to

[8:10:16] us to use. For example, there's going to be recurrent layers, convolutional layers. There's going to be attention layers like in a transformer. The most basic kind of layer that is basically like a perceptron is known as a dense

[8:10:34] layer. So dense is the most basic kind of layer. And we're going to be using dense almost exclusively for now. So what a dense layer is is basically like

[8:10:47] a perceptron. it. A dense layer just takes a weighted sum and passes it through an activation. So it's just a feed forward kind of layer that's just

[8:10:59] doing the most basic thing any neuron could do, which is weighted sum through an activation. So a dense layer is the most basic, but that's not to say there's other kind of layers. There's many different layers that we will see

[8:11:12] when we build our models. As I just said, convolution, attention, recurrent, um, LSTM, all there's all kinds of layers and I'll show us the documentation of, uh, what layers are available, but this is the

[8:11:26] one we're going to focus our attention on is the dense layer. It models just a activation. It models a collection of neurons where every neuron is just a activation. So this is going to be our most basic kind of layer in the inputs

[8:11:43] for a dense layer needs to be number of neurons usually number of neurons neurons usually number of neurons and then the activation function.

[8:11:55] Um so I'll again we're going to see this inside of the code but these are the two things we typically will um declare when we build a dense layer to basically say okay this layer is going to have 20 neurons or it's going to have 10 neurons

[8:12:10] neurons or it's going to have 10 neurons or it's going to have 256 neurons and all of those neurons are going to have relu activation or it's going to be softmax activation or it's going to be sigmoid activation um whatever

[8:12:23] activation that we want to apply. Okay, so uh again that's going to be so uh again that's going to be tf.caris.layers.dense

[8:12:35] vision. We'll then we'll use something like a convolutional layer or we get into RNN recurrent layer or in transformers an attention layer things like that. But right now we'll mostly use dense which is just the most basic

[8:12:50] kind of layer you can have that's just basically short for feed forward which means weighted sum through an activation. Okay. So we need layers and activation. Okay. So we need layers and we also need models. So models will

[8:13:06] we also need models. So models will allow us to connect layers together. So think of the model as the overall network. So, so what we're going to do typically the typical pattern is you declare a model and then add layers to

[8:13:21] declare a model and then add layers to it that will be connected together. Um, and so uh think of think of the the model as our basically our object that contains our neural net. Um, our neural net is the model and then

[8:13:37] within the neural net obviously are all of our layers with our neurons. So those are connected together and then they um generate all those connections by uh

[8:13:49] automatically under the hood whenever we add them to the model. And something that we'll see is that we can we can add them into the model uh sequentially which means that they will be um connected together in that order.

[8:14:03] Meaning like we want this layer first then we want this layer then this layer then this layer. Um we can also do it recursively meaning we basically say here's my starting point here's my ending point and I want you to connect

[8:14:17] everything in between um all the layers that I've declared in between. Um so we will we'll I'll show you both of those ways of building the model but um we

[8:14:29] need typically for any neural net we need an overall model object and then we need an overall model object and then we need layers to be uh to every model has layers inside of that model object. Okay. And then we then we'll just need a

[8:14:43] way to connect those layers together which I'll I'll show you how to do. But the model has some very important functionality to it. Mainly it has these

[8:14:55] two functions which are going to be uh really really critical for us such as the training. So every model has a fit function which executes this just function which executes this just executes the training

[8:15:12] function which now does this look familiar to us? Hopefully it does. Fit and predict. What do you think that's inspired by? That should hopefully look familiar, right? AFIT and a predict. That should look really, really familiar

[8:15:27] to us. And it's actually no surprise that they chose those words specifically. Ffit and predict. Those are directly Ffit and predict. Those are directly inspired by scikitlearn, right? sklearn

[8:15:41] every model there had a dofitit and predict really no different here when we actually just going to be a matter of executing a fit the difference is when we execute that ffit we need to set a bunch of stuff up we need to set up what

[8:15:56] the loss function is we need to set up what our gradient descent algorithm is and and there's a lot of hyperparameters there that are involved in the fitting there that are involved in the fitting of a neural net um so this ffit is a

[8:16:09] little more complex And the the predict is also kind of complex because under the hood what predict is doing is forward propagation right it will generate the output for uh

[8:16:22] whatever input data we pass in. So it's just forward propagation is essentially this predict. Um it also has a convenient evaluate function. Every model has an evaluate function um which uh basically gives us back the

[8:16:40] um which uh basically gives us back the um uh the loss. So we evaluate on a on a collection of data and it it basically um gives us what the loss and any like accuracy metrics that we set up. So what's our precision, recall, accuracy,

[8:16:55] um confusion matrix, those kind of things whenever we apply our model to some data that has labels. So assuming we have labels, we can do an evaluation, we have labels, we can do an evaluation, which is like what is our loss um and

[8:17:10] what is whatever any metrics we declare like accuracy um we can produce those pretty easily. So again, I'm going to show us all this in the code. I'm just trying to orient us right now into the different elements of caris that we

[8:17:23] really care about. So right now we have two which is the layers and the models. We care about those two primarily in order to build and train a neural net. There's going to be other things we care about like declaring a loss function,

[8:17:37] declaring activations, those kind of things. So you know there's there's other small bits that we're going to see along the way. But at the highest level like the two things we care about are the layers in the model that holds the

[8:17:50] layers that holds our neural network layers. Okay. Okay. So there's as I kind layers. Okay. Okay. So there's as I kind of alluded to there's two primary ways of building models which is going to be sequentially or recursively also known

[8:18:04] sequentially or recursively also known as functionally. Um so sequentially is as functionally. Um so sequentially is uh the simpler way of the two um which I think is a really good way to get started with model building um because

[8:18:17] started with model building um because it allows you to create models basically layer after layer after layer in a linear sequence. So what that means is you basically um declare that I want you basically build a list and say in this

[8:18:33] list this layer goes first then this layer then this layer then this layer which for some models makes a lot of sense like if it's a very simple model to say okay I have this layer then this layer then this layer and it's going to

[8:18:49] go forward through that sequence right from layer after layer after layer which is very natural. model that makes sense. So the sequential way of building a model is kind of the simplest way to do it. Um but the recursive way to do it or

[8:19:05] what I should say the functional way to do it is um a little bit more flexible in the sense that um it allows you to have layers that can be uh shared and

[8:19:17] nonsequential. So for like recurrent nets that's really important. So what we're going to see is I'm going to show you examples of both. We're going to going to build a model using the functional API

[8:19:32] um so that we can get exposed to both. But most of the models we'll do to begin sequentially because they're actually going to be very simple. Um it's not until we start building some more complex models that the functional will

[8:19:47] be actually better for that case. But the the functional um allows us to have the the functional um allows us to have um kind of exotic structure uh because

[8:19:59] all we need to do is basically say how one layer acts on another layer and and we can connect all those kind of recursively. And so it gives you a chance to do things nonssequential because you can have a layer that's

[8:20:13] further down the line act on an earlier layer by um having like a recurrent connection back to the previous layers and you can do that in the functional API uh when you cannot do that in the sequential. So I'm going to I'm going to

[8:20:28] show us both ways. Both are widely used. I think both like when you're building something relatively simple, I think sequential is the easiest for a reader sequential is the easiest for a reader to understand and see. Um, but if it's

[8:20:41] always going to be using the functional way to do it. Uh, because you have to sequential because you're not going to have a flow that's just directly in feed

[8:20:54] forward. Sequential is basically only for feed forward where things are guaranteed to go from one to one to one to one to one and just go forward which the case. That's all we really care about in the beginning. But as we get to

[8:21:08] that functional way of doing it. Let me give you an example. So let me give you an example what the sequential um uh way of building a model looks like. And this is actually a very prototypical case where you build your

[8:21:25] model as a sequential object. So you can see directly here we we initialize a sequential model using this sequential object here. And look at what we pass in

[8:21:39] is a list. So inside of the sequential is literally a list of layers. That's that's what the sequential model takes. And the it's it's purposely a list

[8:21:51] And the it's it's purposely a list because what the list signifies is this layer should be first and then this layer should follow it, which is exactly what we've seen in our drawings, right? Like there's a layer that's first on the

[8:22:06] left and then that flows into a layer on the right. So it's it's it's sequential. It's this layer and then this layer and then one more layer and then one more layer. Right? So, so sequential is just kind of for exactly those use cases

[8:22:22] we've been drawing. What I want you to see here is an interesting uh very simple model, but what it's saying is here's our dense. So, here's where we're using that layer. So, we have a list of layers. Here's that dense,

[8:22:37] which is the most basic kind of layer we can have. What that means is I have a bunch of neurons. How many neurons do I have? I have 64, which is a choice. So I can choose how many neurons I want in that layer. So

[8:22:52] we're declaring 64 neurons to be modeled in that layer. And what we're saying here is there's an input shape, meaning the input data to this initial layer is

[8:23:05] this size. So this is just orienting the network to say my input layer is going to be this shape. So I should expect data to flow into this network that's this exact shape. Um so that's however many dimensions it is which would be

[8:23:21] like however many features we have. For example like if we had 20 features that input dimension would be 20. Um but this is saying we have 64 neurons and the activation of every one of those neurons is a relu activation which is pretty

[8:23:36] standard right a relu activation is pretty common. And then here now what pretty common. And then here now what this is signifying is this layer this is signifying is this layer flows into this layer because it is next

[8:23:50] in the list. So the outputs of that layer flow into this layer which has 10 neurons and a softmax activation. Now what does that mean? That means that we are likely predicting

[8:24:07] 10 categories. Um because I have 10 outputs which are softmax activated. So that means they are all going to be probabilities. And not only that, they're all going to be probabilities that total up to one. So this is likely

[8:24:26] this is the and by the way, this is the output layer because it's last in the output layer because it's last in the sequence in the list. It is last. So it is the output. There's nothing else. It's not connected to anything else.

[8:24:39] It's not connected to anything else. Right? So this has to be so the last Right? So this has to be so the last layer in the sequence nothing else beyond it. Last layer equals the output. And so you can

[8:24:56] actually see this is a model. This is going to be a neural net that is designed to predict 10 different classes. It's providing us a probability for 10 different classes. So for example, this could be used for

[8:25:10] example, this could be used for something like the MNEST data, like that where there's the the digits 0 to 9 which are 10 different categories,

[8:25:22] right? That could be used for something like that. Um I'm just throwing that out as an example, but hopefully um hopefully this makes sense. What you're hopefully this makes sense. What you're seeing here is two layers. You have

[8:25:36] really this initial uh hidden layer with 64 neurons and then uh hidden layer with 64 neurons and then you have a output layer that is size 10 with a softmax activation meaning you're going to produce 10

[8:25:50] going to produce 10 10 probabilities. So the so it's a multi this is likely a network that's used for multiclass classification.

[8:26:04] network, right? This is our neural net. This here is our neural net object. Look how easy that is to create in caris. It's just a sequential with some

[8:26:16] caris. It's just a sequential with some layers. That's really all it is is is a sequential and a bunch of layers. Okay, let me show you the difference with the functional API and and you you can see the difference, but hopefully it makes

[8:26:28] sense. Now, this is the exact same network. It's just built in the functional API. Exact same network, though. It's just going to look a little different. So, what we do in the

[8:26:42] functional API is we actually are going to declare an input layer as the very beginning. So, we explicitly define an input layer, which is this. Same thing

[8:26:55] though it has a particular shape but notice it's there's an input layer we're explicitly defining in the functional and the reason we're doing that is because when we build a model we need we need a layer to tell it where to start.

[8:27:09] We basically need a starting and stopping point. Um so so this first layer is our starting point here. So we start here with this input layer. Now start here with this input layer. Now what I want you to see is we now declare

[8:27:24] what I want you to see is we now declare um a dense 64 activation but look at the um a dense 64 activation but look at the notation of the code. We are basically applying this layer almost like a function right almost exactly like a

[8:27:36] function right almost exactly like a function. What are we applying it to? The input layer. That's really critical in the functional setup because what this is saying is I want to take this layer and apply it to this layer which

[8:27:53] layer and apply it to this layer which is the input layer. Now if you look at is the input layer. Now if you look at the next layer, this is our dense uh 10 with the softmax. So this is our output. What are we applying that to? We're

[8:28:06] applying this to our hidden layer which is what we declared here. So, do you is what we declared here. So, do you guys see how this is more functionally guys see how this is more functionally defining how layers act on each other?

[8:28:19] This is rather than building it in sequence, we're saying, okay, this layer sequence, we're saying, okay, this layer acts on this and then this layer acts on this. All we need to do then to build the model is to instantiate a generic

[8:28:32] the model is to instantiate a generic model and tell us where the starting and stopping point is. And what it will do is connect everything together because we've already declared how they all act on each other. So what we're doing here

[8:28:46] on each other. So what we're doing here is saying start here should be here. And because we've specified how everything acts on each

[8:28:58] other, this model is going to recursively go through and build the graph because it knows how everything acts on each other. It knows that the input layer is an input to this and it knows that the hidden layer is an input

[8:29:13] knows that the hidden layer is an input to this. So it's going to connect all those together in this model here. So it's a it's a different way of declaring it, but it's the same exact model, but this is just the functional way of doing

[8:29:27] this is just the functional way of doing this. Now you might you might ask why would we do it this way rather than the sequential? It's because you now have an opportunity to build more complex architectures in this in this way of

[8:29:41] coding it. You have an opportunity to build in a much more complex architecture because what you can do is you can have things be shared. You can basically say I want to take one layer and add it to another layer um like plus

[8:29:57] another layer. um you can do things like that which in more complex architectures that later on when we get into like convolutions and recurrent nets. Okay.

[8:30:09] Okay. So just to compare and contrast again, the sequential is mainly for a linear stack of layers which is literally in that list usually because of that it limits itself to simple models, limited customization, very

[8:30:23] simple models that you can get up and running again which is going to be like in the beginning of when we're learning this. That's mostly when it's going to this. That's mostly when it's going to be um applicable. Um but the the

[8:30:36] functional has very high degree of customization which lends itself well to complex architectures um which is going to be useful down the road when we get into those kind of networks. I think it's good to know both

[8:30:49] sure. >> And that brings us to the end of this this video, we started from the fundamentals of deep learning and move step up into neural networks, artificial neurons, perceptrons, activation

[8:31:03] function, loss function, gradient design, and back propagation. We will also explore how to build and train models using TensorFlow, Keras and PyTorch, and how these frameworks help us solve real classification problems.

[8:31:16] The key takeaway is simple. Deep learning may look complex at first, but once you start understanding neurons, layers, laws, and optimization, you can start building powerful AI models with confidence. If you found this video

[8:31:28] subscribe to Simply Learn for more such machine learning, deep learning, and the latest tech skill. Thank you for watching, and see you in the next session.