[00:02] problems using computer vision, recommendation system, forecasting models and AI agents, that is exactly what this capstone session is all about. simply learn. In this video, we will explore the Microsoft AI engineer
[00:17] capstone and understand how learners can apply everything they have studied in the program to build practical AI solutions. This topic is important because AI engineering is not just about learning concepts. It's about applying
[00:29] problems. Whether it's a detecting objects for autonomous driving, recommending tourist places, forecasting restaurant sales, or even building multi- aent systems, Capstone projects help learners connect theory with
[00:43] session, we will cover the three major Capstone projects options. First, we will look into the autonomous driving project where learners use object detection while yolo and analyze self-driving accident data. Next, we'll
[00:56] explore the tourist project where learners classify landmark images using recommendation system using collaborative filtering. Then, we'll be covering the sales forecasting project where learners analyze restaurant sales
[01:09] compare the regression models, and forecast the future demand. We'll also discuss key technical concepts such as TensorFlow, Keras, transfer learning, image augmentation, pandas, group by data, merging, recommendation system,
[01:23] regression models, and time series feature engineering. Towards the end, we will also move into agentic AI and understand how frameworks like langraph, autogen, and QAI help build AI agents that can plan, use tools, complete the
[01:35] task, and work together. We'll also look at crew AI travel itinerary demo where multiple agents collaborate to create a complete travel plan. By the end of this approach your AI engineer capstone projects and how modern AI engineering
[01:49] is moving beyond simple models into intelligent tool using AI systems. share you something exciting for everyone who wants to start a career in AI. Microsoft's AI engineering program is a beginner-friendly program designed
[02:04] to help you understand the complete AI stack from Python, data science, machine learning, and deep learning to generative AI, NLP, Agentic AI, and Microsoft AI tools. The program includes 140 plus hours of live training,
[02:17] hands-on learning with tools like Azure, OpenAI, Copilot, Studio, Autogen, PyTorch, TensorFlow, Lang Chain, Hugging Face, and more. plus 12 plus real world projects and three capstone to help you build practical AI skills. You also get
[02:32] Microsoft and simply learn access to Microsoft learn certificates and preparation support for Microsoft Azure AI 900 certification exam. So if you want to move beyond just using AI tools
[02:44] and actually learn how AI solutions are built, developed and applied in real world scenarios, this program is worth checking out. The link is given in the comments. Go check it out. Before we move ahead, here's a quick question for
[02:57] you to answer. Which AI framework is mainly used to build a multi- aent travel planning system? Is it Crew AI, Excel, Tableau, or is it PowerPoint? started. >> Okay, so the the very first so again
[03:12] three projects that we'll go through and these are the options that you have to these are the options that you have to uh work on. Um and so the this very uh work on. Um and so the this very first one is on autonomous driving. It's
[03:27] centered around this idea. Um and so there'll be each project has two parts there'll be each project has two parts to it. Um so and uh so we'll go through both of those parts for this but the theme of this first project is centered
[03:41] theme of this first project is centered around autonomous driving. Um so this is project one. Uh and again you should have in the capstone course you should have these reference materials that have the uh uh notes or the problem
[03:55] description and then the data sets that are relevant to the project. Um so again with you guys so you have a little bit more background on them and then also
[04:07] more background on them and then also review some some relevant um examples that I think will help you with these particular projects. Um but let me describe the autonomous vehicle one.
[04:19] vehicle one. So um the first uh part is centered on object detection, right? So, we've worked on object detection before, but
[04:31] um it has to deal with, you know, what a typical autonomous vehicle system would have to do, which is obviously detect vehicles on the road um uh at, you know, vehicles on the road um uh at, you know, for tracking uh purposes and driving
[04:46] for tracking uh purposes and driving purposes, obviously. Um purposes, obviously. Um and uh this uh part of this data that twofold. one, it's going to be a bunch of images that we can run object
[04:59] detection on to detect what kinds of vehicles that we are uh seeing in the image. So, we'll there'll be a part one that does that and then part two to this that does that and then part two to this project will be analyzing some data um
[05:12] related to accidents with self-driving cars. So, there'll be kind of a a data frame style data set and you'll do some analysis. we'll talk about what that analysis. we'll talk about what that looks like um you know related to uh
[05:26] looks like um you know related to uh that data set. So the first part is doing object detection that can um predict the vehicle type. So there'll be different models of vehicles um as well as drawing
[05:41] models of vehicles um as well as drawing you know f producing that box that uh we typically do for object detection right that um locates the object within the that um locates the object within the image. Um and so uh one of the things I
[05:54] want to do is go back and look at that uh example that we worked on with YOLO and just to refresh us a little bit on that process. Um, but in this uh
[06:07] project, you'll have some data that that mimics kind of the example that we went through back then. You know, you'll have images and then you'll have those files that have the um ground truth kind of box coordinates and box label of what
[06:22] box coordinates and box label of what vehicle it is. Um and so you know if we have that data we should be able to follow mostly the same steps that we used when we uh went through you know building and and training the YOLO model
[06:37] and then using it for prediction. So I do want to walk through that uh example really really relevant for this pretty much identical to doing this just with different data right. So this will be um predicting different types of vehicles
[06:52] that are in in and drawing the box around the vehicles within these images. Um so that's one part uh of it. The second so that's one part uh of it. The second part is um analyze the uh autopilot
[07:06] program and some of the accident data that I talked about. Um so there'll be a bunch of different tasks related to that that we'll go through. So, uh, part one has images that look like this. So, uh, this is part of your
[07:20] data set for this project is the images.zip, uh, folder that has all of images.zip, uh, folder that has all of these image files in them. Um, these image files in them. Um, so you can see they're just uh images of
[07:33] of them are buses, different types of vehicles. Again, our goal is to be able to train and utilize an object detection model, something like YOLO that we did,
[07:45] you know, to be able to draw those boxes around these guys and and predict what kind of vehicle it is. Um, so you'll be given data that kind of uh resembles exactly that kind of data set we used back in in in that example.
[08:02] Um, so you have images like that. Uh so then uh it's all about uh going through and building the object detection model. So the first task really the first part of the project is um building that object
[08:17] the project is um building that object detection. Um I know it says here create detection. Um I know it says here create a CNN architecture. Um but in reality a CNN architecture. Um but in reality you can follow the uh steps of using a
[08:29] pre-existing model like YOLO. Um it it wouldn't really make a lot of sense to wouldn't really make a lot of sense to do it by scratch. um and and not use an existing model that's out there like YOLO. Um so that's again why I think
[08:42] it'll be relevant to go back and look at that again. But that would be the advice here is just use an existing one and we'll train it with this data. So we won't create one from from scratch. We don't need to do that. We can just use
[08:54] an existing one and and train it against this data which is that whole transfer learning process, right? So taking a model that's already been trained off the shelf and then just updating it with our own data in in this uh example.
[09:10] So um again I'll show us how to do that. I'll go back and review that and it'll be really really relevant to this essentially similar task.
[09:24] Okay. So the goal is to train it and then um basically run that model after it's been trained on some of the test images to see if we can uh to see what the detection results look like um which we also did in that example back then.
[09:37] Right? So we we first trained it which was the transfer learning process. Then we applied that model to uh then we applied that model to some example images to see what kind of boxes it was drawing. Right?
[09:52] And so again, I I'll go through that today. So we have some review on on today. So we have some review on on that. Um but but yeah, that's the first part of it is just doing object detection with this data.
[10:05] detection with this data. Um so that's the first part. Then the uh well, let me pause there. Any questions on that first part? Again, I think the deep learning lesson 10 that we did on object detection will be really really
[10:19] relevant to that. pretty much doing if you choose to do this project. I think it'll be pretty much doing that again and we'll go through that today just to review it, but I think it'll be fairly close to following that.
[10:41] good. So, let's talk about part two of this first one. So um this uh part two will be dealing with this CSV. So part of again you'll have the images and
[10:53] their labels and then you'll also have this uh CSV for this part two which has a bunch of um data centered around accident cases. accident cases. Um and so there'll be some text
[11:08] description, there'll be some statistics about them. um and whether or not autopilot was being used. Um you know, how many occupants there were, if there how many occupants there were, if there were any deaths. Um so, a bunch of
[11:23] accident data here. And our goal will be to do a bunch of uh analysis with this. Um you can see a little bit about the description of the column. So, um that's always helpful to know is kind of what each column represents. And so, you have
[11:38] some information here about that. Um, so, uh, this is, you know, just some information about those columns. So, you can see a bunch of, uh, different information here. Um,
[11:52] including the model of the car, if insurance has been claimed, total number of verified deaths, those kind of things. number of vehicles that were things. number of vehicles that were collided with, if any, um number of
[12:06] with, which would be again if there were any recorded. Um that's something to be aware of with this data is there are, you know, it can be null, which means there was no data for that. Um so if you're doing this project and working
[12:21] with this data, be careful with the NLES um in terms of filling them in. Um, and you're I don't think you're building any model with it necessarily. Um, you're not building any model with it. It's just uh for analysis purposes, it
[12:37] may be easier to fill in all those blanks with a zero. Essentially, maybe something you want to do. Um, but that's just a suggestion. Yeah, it's from a Yeah, it is legit. It's from a sort. You can see like there
[12:52] are rows that have the case sources, which are information about these accidents. Um yes this is it's it's old data it's from 2022 most of these right so you can see in the year
[13:07] um yeah it's older but
[13:31] with it is some initial inspection, checking for data types. Again, the probably the biggest task here is dealing with the missing values. Um, in which case, my recommendation there would be likely to fill in with zero.
[13:46] There's basically no information about it. So, we could put a zero in there. um it. So, we could put a zero in there. um which I think would be fine. Uh and um So, you know, there's probably columns in there that aren't going to be that
[14:02] relevant like the case number, um an ID, those kind of things. Uh um you know, you can feel free to drop them if you want to uh just to make the data
[14:14] set a little bit smaller to to work with. Um, but part of the exploration you're going to be doing for this is, uh, you know, figuring out how many events there were by date, per year, and per day for each
[14:32] state and country. So part of what I would recommend there is building um it you know it may be helpful to build some images or sorry some visualizations for this such as doing different plots like box plots that can
[14:47] uh use some of these fields like country and state um as like the x-axis and and count how many uh accidents there were, how many uh deaths there were and those
[14:59] kind of things. Um so uh you can also another thing that may help you with this is doing group by. So we can't forget like part of our aggregations that we can do on dataf frames is grouping by uh different
[15:16] columns right so we can group by the country we can group by the state um we country we can group by the state um we can group by year um and then do totals like a count of how many events there were. That's kind of what this this
[15:30] first one is. This part a is is likely just a simple group by um for these things. And then uh you can do you can aggregate like a count of how many
[15:42] events there. Basically every row is an event, right? Some type of event in the data. Um whether that's like a basic accident or a deadly one. Um they're all
[15:55] they're all some type of accident event. Uh, so that's probably the easiest way to go, but you can also go like a visualization route for that too. Um,
[16:07] yeah, that's yeah, it can be. Yeah. Yeah, definitely. Can the victim be the driver? Yes, there's a column for that uh in the data.
[16:22] the driver, so Tesla driver, there's a column for that. Yeah.
[16:35] exploration is analyzing the death events in particular. So this would be basically filtering the data frame right to uh every row where there was at least to uh every row where there was at least one death. Um so filtering it to that
[16:49] and then kind of analyzing that data frame that you get. Um so what is the number of deaths in each accident? um we just kind of um you know like an average uh essentially um how many of them had the Tesla driver so it's a count of how
[17:06] many there are with the driver death um what is the fraction of events in which one or more occupants died what is the distribution of events in which the vehicle hit a cyclist so what is that
[17:18] you can even you can even do a value counts there which gives you that um and these are all columns right so the Tesla the driver's death is a the Tesla the driver's death is a column. Uh occupants uh is a column. And
[17:31] column. Uh occupants uh is a column. And then the uh the um uh driver we said is a column. So you're just doing different value counts on columns. You could also just do a histogram, right? You could that can
[17:43] histogram, right? You could that can also be a visual of that uh um also be a visual of that uh um distribution as well. Um and then there's this is another kind of filter. How many times has the
[17:55] accident involved the death of an occupant or driver along with a cyclist? Um, so that's kind of a filter for did we have a death in this column or a death in this column? How many rows do we have that match that filter? Um,
[18:12] and then how many times did we have Tesla colliding with other vehicles, which is another column, right? So there's there's a column specifically there's there's a column specifically for that. Um which is other vehicle. Um
[18:26] so um this analysis is mostly just looking at value counts and and histograms for these various columns. So fairly straightforward I would say for fairly straightforward I would say for this um to to do this analysis there.
[18:43] Um then you can group by the model uh and look at the counts per model. So that that again could be a very simple graph like a um uh a plot with Seabor could help you there. Um it could also be just doing a
[18:59] group by um and then doing the value counts um once you group by the model there's lots of different approaches I way to get this, you know, get that relevant data and um
[19:15] maybe even build a visualization which I think would would would be great. Um and then what is the histogram or the value counts of the autopilot? So that's there's a column for autopilot. If it's a one, um that means autopilot was being
[19:30] used. So that that's going to be kind of a filter down to how many rows had autopilot of one. So it's a very simple question that can be answered from the data frame. Uh so that shouldn't be too bad. So a lot of these correspond to
[19:45] existing columns. It's not um not too much. It's just interpreting those and kind of using value counts using a histogram to get the distributions of them. Okay.
[20:04] so this is the first this is the first project one of three. Um it deals with object detection and then doing some basic data analysis of of this um event
[20:16] basic data analysis of of this um event that uh accident event data. Um any questions surrounding this? I'm gonna I'm going to do object detection uh examples um or just walk through the I mean the
[20:31] already did before. So I'm going to go back through that um after we go through the other projects. So we'll we'll come back and do that. So we'll we'll come back and do that. But any questions on this?
[20:54] good, good, good. So, let me go to So, detection. So, that that'll be very, very relevant to this. Um, you know, the other parts that are relevant to this are likely in our applied data science
[21:08] at yesterday with data wrangling, right? So, doing those kind of filters and So, doing those kind of filters and group buys. Um that was also in in uh group buys. Um that was also in in uh lesson four of that course. Um you know
[21:23] lesson four of that course of the applied data science and also lesson 10 on feature engineering probably have some relevant stuff related to that data some relevant stuff related to that data analysis. Um but the the deep learning
[21:35] course lesson 10 is going to be really valuable for object detection for that valuable for object detection for that first part
[21:49] is there's three different projects. Each one has two parts. So, we just went through the first project which has two parts. Part one is the object detection for the vehicles and then part two is data science data analysis on the uh on
[22:05] the accident data. Both of those parts just are are for the first project only. So again, you're just going to choose one of the three to do. So if you choose
[22:17] project one, which is this autopilot themed uh um project, you would be doing both parts. So you'd be doing both of those. You'd be doing the object detection and the uh data analysis, which is uh all two parts for the for
[22:33] which is uh all two parts for the for the first project. need to choose one of the three projects. It's just each one has two projects. It's just each one has two parts.
[22:51] uh to the second uh project that you have available.
[23:15] related uh basically enhancing tourism with AI related project. So if you're interested in that, this would be a good one. Um so uh in in the background for
[23:28] this second project is there you know there are centuries old historical structures that preserve a countries and communities history to improve tourism. advanced machine learning techniques to monitor the condition of these
[23:41] structures and report if any of them need maintenance. Um also understanding critical for effective marketing and excellent way to supplement existing marketing outreach.
[23:56] So this project is going to center on two parts as as just as the first one two parts as as just as the first one had two parts. Part one is going to be a problem. So you're going to be looking at different images of structures and
[24:10] at different images of structures and classifying them. Um and then the uh the the second part is going to be building a recommendation system using collaborative filtering um to recommend a tourism destination
[24:26] a tourism destination um given a location. So, sort of like um item recommendation, kind of like what we've talked about in the past where, you know, if you're browsing a website and you're on a product, you see
[24:38] recommendations for other products. Um or like if you're looking at a movie, or like if you're looking at a movie, you see other movies. Um same kind of location, here's other relevant uh landmarks or locations that are
[24:51] landmarks or locations that are recommend recommended to visit. Um so, so that'll be the two parts. is the recommendation part and the uh image recognition part. So we'll break those down and talk about each one of those,
[25:04] but those are those are the two main parts for this one. parts for this one. So part one is uh building a uh as I said a computer vision model that can uh predict the category of a structure. So
[25:19] there'll be various uh there'll be images, we'll take a look at them that have different types of structures like bridges, uh statues, um different landmarks, and you'll be uh basically categorizing those into the
[25:33] different labels. Um and you know, that's a problem we've done many times is given an image, here's the label for it. That's one that we should feel comfortable with, but I'll go back and review how to do that effectively with
[25:47] transfer learning. Right? So, we've um uh you know talked about transfer examples with it. I'll go back and and do that as review for this because I think it'll be really relevant to this part one is doing transfer learning with
[26:02] uh image recognition. So, we'll go back and do that. Um and recommendation. I'll go back and go through that too because it's been a while since we've looked at that. Um so so those two things uh transfer learning
[26:18] um I think is lesson nine in the deep learning course that'll be really really learning course that'll be really really relevant for part one and then um our I think lesson seven in our machine learning course on recommendation
[26:31] learning course on recommendation uh will be um uh will be um uh uh will be you know the lesson seven will be really relevant for machine learning.
[26:58] recognition and I think um the best way to approach it will be to do transfer learning for it. So, I'll talk about that, but this is one of those examples kind of like the flowers uh data set that we had or you know that had
[27:11] different folders for different uh categories. So, they had like remember we had like roses, tulips, dandelions, whatever it was in those different folders. Um here we have uh different structures
[27:28] which have images in these different folders. So we have the um these are just different structures uh um stained glass domes column tower um flying
[27:40] buttress um you know these are all just different images there's images within structures um and our goal is to you know build a um and our goal is to you know build a model that would be predicting um one of
[27:55] these labels for the for a structure given an image right so we have an image it should be one of these categories of structures. Uh and so we want to build structures. Uh and so we want to build our model to predict that. Um
[28:09] and there's also a test set of so there's uh um a zip folder that has uh uh you know uh basically training images and test images of of these guys. Um,
[28:23] again, I'll go back and review and show you the relevant notebook that that you should use as a guide for doing this. I think it'll be um it'll follow very closely to like our transfer and learning examples that we've done. Um,
[28:37] where especially like that example where we had different flowers, remember that were in these folders and we can load those into a nice caris data set that splits them into training and tests and all that. Um so we have practiced before
[28:53] how to do this and so uh I think it'll be pretty straightforward to follow those follow the examples. I'll just remind us of that uh today and go remind us of that uh today and go through those. But
[29:05] um the first task again is related to those images. What we want to do is um do some sample plotting. Now this we have examples of doing two. We haven't focused on doing that too much but we have some code that I'll review today.
[29:19] that can help us um basically plot some of these uh some of these images. And so that's not too hard to do. We can load in those files and then uh easily generate a plot of them. We we've been able to do that before. That's not too
[29:35] able to do that before. That's not too hard to do. Um and then the the more interesting part obviously is uh building a model. So this is where you want to um use transfer learning, right? So I would not advise
[29:52] uh building it from scratch even though you could, you know, you could um try to put together different convolutional layers and build out your own network. I would recommend using an existing network off the shelf and doing transfer
[30:06] learning, which is what we've done in the past, right? So we have practiced using one of those models like VGG16 or ResNet loading that in from caris and then remember what we did is we left off the the last few layers the top of it is
[30:23] the the last few layers the top of it is what it's called um and then we inserted our own layers uh there and then just retrained it on the data. Right? So I don't think it makes sense to build your own from scratch. uh it's not going to
[30:37] pro it's likely not to be as effective as one of those existing architectures as one of those existing architectures which are known to be pretty good. Um which are known to be pretty good. Um so again I'll show us that example. Uh
[31:00] that's true. We did not we skipped over that. Um, so we need to make those up. Uh, essentially we skipped over those. So I can uh message offline after this or at the end and clarify that. But um, yeah,
[31:18] we because we need a Microsoft certified trainer to administer that. Um, which is not me. So we we skipped over those in favor of just continuing our
[31:30] those in favor of just continuing our generative AI material and uh review. So generative AI material and uh review. So we do need to to circle back to that. It shows completed because in place of that we covered that other material
[31:45] but definitely owe it owe those those courses.
[32:03] notes it has a hint to do transfer learning. So of course um that's what I learning. So of course um that's what I would recommend doing and we will um uh we will practice and review what we did back then to do that just so we see
[32:18] did back then to do that just so we see it one more time. Um, so and it even says here, uh, modify the top of this network by adding whatever dense layers you want with the activation and then using dropout if you want. Um,
[32:32] so it's going to be this the same exact situation as that example we did. Pull situation as that example we did. Pull in a model off the shelf. Um, you know, uh, leave off the top of it, insert your own layers, and then train that network
[32:45] own layers, and then train that network against the the data that we have. uh here in this project. So, and then we know like in order to train it once we have that architecture like pulled off the shelf and we've
[32:58] layers, it's just a matter of setting up the training, right? So, we do the usual things. We set up our uh loss function, our optimizer, which is going to be atom, our metric usually accuracy. Um we can even set a early stopping call back,
[33:14] which is what this suggests to do. Um so and we that's something we've done in and we that's something we've done in the past and we can review that too. Um and then we can uh basically do the model.fit right so we can uh uh uh train
[33:29] model.fit right so we can uh uh uh train the model um uh by running the model.fitit once we have everything set up. So so that part is fairly easy um getting everything set up from the transform learning perspective that uh
[33:43] transform learning perspective that uh is worth reviewing and practicing. is worth reviewing and practicing. We'll do that. to do too in this project which uh is interesting is training the model twice.
[34:00] interesting is training the model twice. Um, one time without augmentation layers in the front. Um which so remember augmentation is uh those additional layers we put in the very front to generate variations on our on our images
[34:16] like doing the zoom or brightness change or flips or rotations. Um and so uh it wants you to do training twice. So basically an architecture that has that at the very front um and one
[34:32] it's easier you just you don't have to add anything additional to the transfer learning but with it you want to add in the um you know the transformations and we can review that too like how do you
[34:45] do that um but it's essentially um transformations that we want to have and then just putting that in the in a sequential putting that first, putting the transfer
[35:00] learning model after that with our own custom layers of course, but you know really having that augmentation up front. Um uh having that augmentation up front and so we can do it with and without that.
[35:14] And the purpose of doing it with and without it is to see the difference like what kind of difference does it make to have that additional variation? And the idea is like um it should help us generalize, right? If we're able to um
[35:29] train our model on those different variations of the input images, we should do a better job of predicting generally against the test set. Um so It's supposed to help us with generalization to use augmentation, but
[35:44] we can see the difference in the stats, right? So we should be able to train it with and without the augmentation to see the difference. the difference. Okay. So that is the first part. Uh
[35:59] again the resources that are going to help you a lot if you're working on this one are the transfer learning uh which I are the transfer learning uh which I think is lesson nine uh or 10 nine or 10
[36:11] from I think it might actually be 10 from uh deep learning. um and we'll we'll review that today. So, another one on our list. So, we'll do um object detection, we'll do recommendation systems, and we'll do transfer learning.
[36:28] systems, and we'll do transfer learning. But any questions on this part of it?
[36:40] very good. So, let's go to part two. So that's part one, image recognition with those different structures again using transfer learning mostly. Um let's go to part two which is going to be building a recommendation system. And so as you
[36:54] would expect we have uh different data sets here that represent the users and the items um that we would build the recommendation system off of. So we have recommendation system off of. So we have um users uh uh in this user CSV and then
[37:09] um users uh uh in this user CSV and then we have um basically uh items in this tourism with ID and the main the the products here we have to think of the products a little differently than usual. They're not physical products,
[37:23] right? They're and they're not movies. In the examples that we've done in the past, we've mainly worked with movies or like products, right? Here the product is the location. So it is like these places. So there's a
[37:38] place ID and a place name like this monument or this uh you know museum or whatever it is. These are different places we could recommend for tourists to visit. And so they have different item ids just like any other product
[37:55] would in a typical data set. Right? So we have users products and which are tourism locations and they they have ratings that are given to them. ratings that are given to them. Um and so uh we want to use all of that
[38:10] information to build a uh collaborative filtering system to to you know given one of these locations to then recommend other similar you know locations that other similar you know locations that that may be worth visiting.
[38:25] that may be worth visiting. That's the goal. Um then there's one more data set which is the uh combination of the users and the uh the items and their ratings. So which of course is the data we really care
[38:40] of course is the data we really care about is is the um users and these these different places. So we should think about these are our items essentially right these are our items these are our
[38:53] these are our items these are our ratings for those items. So it's th this ratings for those items. So it's th this um part of the project is going to be very very similar to the recommendation systems that we built um back when we
[39:07] were doing like movies or anime. Remember we had those anime shows and recommendation. It's going to be very similar. It's just you know these these product IDs are now um tourism locations that we're
[39:21] recommending. Okay. So, and we'll review that, right? So, we'll that's that's the point of this session and really um next Monday's session is to just have some time to
[39:33] session is to just have some time to review these things as you guys are uh you know um working on on these final projects here.
[39:52] So uh there there's a description of each of these data sets. So we have the you know user CSV um which we already talked about just contains information talked about just contains information about the uh user age and user ID that
[40:05] may not be very relevant for building a collaborative filtering system but it but it's there. Um this is more metadata about the places. So obviously the place ID is really important. Um, but given the place ID,
[40:19] we can use this data to kind of look up what it's called, what a description of it, the actual like city and country location, um, latitude, longitude. Um,
[40:31] so this is more information about those products. This is the data that we're really going to zero in on though to build the recommendation system, right? Is this tourism rating CSV. That's probably the
[40:43] most important data is is the actual ratings that were given to these places from users. Um which are you know you know users we have to frame it correctly know users we have to frame it correctly in this users are tourists right so they
[40:57] in this users are tourists right so they are various tourists um and and items are places right so just like when we were doing users in movies or shows here were doing users in movies or shows here we have tourists and their locations. So
[41:12] but and but they have ratings that they've given to these locations. Um so we can certainly build a recommendation system.
[41:25] Okay. So um couple things to do for this part So um couple things to do for this part two is first do some preliminary loading of the data and inspection. So check for any missing values. So you want to load
[41:37] each one of those CSVs as different data frames and check for missing values or duplicates, right? U and then uh basically remove any um
[41:50] uh maybe drop any missing values or uh any duplicates we can drop. Um it says remove any anomalies. So, you know, it may be worthwhile to see if there's any outliers um from the ratings. Uh so,
[42:08] that may be worthwhile to do is see if there's any really um you know, far [snorts] away from the average rating ratings. Um and we can consider dropping those rows if they're really significant outliers. Um so, uh
[42:23] and you can you can use your own technique to find outliers. So obviously we reviewed a little bit of that yesterday uh Monday. Various ways you can find outliers which would be you know doing like the zcore for the rating
[42:36] figuring out how far away it is from the average and dropping any that are average and dropping any that are significantly far away. Um so just do a little bit of that. Um so then we can analyze the ages of
[42:52] the users. Uh so we can look at the histogram of the age from the user CSV. histogram of the age from the user CSV. Um and then identify the places where these users are are visiting. Um in order to do this by the way um you
[43:09] likely want to use that second data set. So, you want to use uh um you likely So, you want to use uh um you likely want to in fact you likely want to um likely want to merge together potentially this this user data set with
[43:25] potentially this this user data set with this tourism uh data set um where you can merge them together with the place ID essentially being like the location. Um, and so, uh, that may be something you
[43:40] want to do is merge those together so you can answer some of the these questions that relate to the user. May also be able to just use this third data set which has users and their places and their place ratings.
[43:53] But, but the goal is there to just answer that basic question. Um, where are the users visiting the most? Uh, so what are kind of the top places that users are actually visiting? um you know they might have given them low ratings
[44:08] but they were visiting them. So what are those place ids um that that have the those place ids um that that have the largest count? So basically value counts on the place ids um maybe a way to go there.
[44:26] which is going to come from that second data set? uh what kind of tourism uh uh each location is most famous for um what city would be the best for a nature
[44:39] city would be the best for a nature enthusiast. So these are going to be um you know probably all information we can get from this data set. Um and uh there's various ways to do that like for instance the nature you may
[44:54] look for that string in this description. So you can see is the word description. So you can see is the word nature in this column. Um or you know that's not hard to do by doing a simple filter. Um and so you you know feel free
[45:09] to use that description column and look for various substrings in there um such as nature. That may be something good to answer that question.
[45:25] four is uh create a combined data set with places and their user ratings. Um which would be like merging the second and third data sets that have the user
[45:37] ID, place ID. If you think about it, we should be able to merge that together because this third data set has user ID and place ID along with a rating. But we have a data set that has information about every single place ID. So we
[45:51] should be able to merge that second and third data set together um to have uh you know essentially a combined data set. Uh so that that'll be a good practice for merging uh which we looked at on Monday right um doing a merging on
[46:09] at on Monday right um doing a merging on a place ID for example. figure out uh what cities had the highest ratings because the place ids
[46:21] will be joined together with the ratings. So once you join that second and third data set you can ask you can answer a lot of questions. Which city had the highest ratings uh on average? Um
[46:35] uh which category of places are you have the highest ratings? Again, those are only things we can really answer once we've merged the second and third data sets. Okay, so basic analysis there. Then is
[46:50] the fun part is a recommendation system, right? So then we can do we can use that third data set to build a a collaborative filtering model because we have the place ID, the user ID um and the uh
[47:04] the uh uh ratings. So the idea is like given a uh ratings. So the idea is like given a current place ID um can we recommend other similar places to visit? Um so this the collaborative
[47:20] filtering we want to do here is going to be similar to the examples we did in our be similar to the examples we did in our lesson that are uh it's not userbased because we're not given a user but it's given a location. So, it's actually
[47:32] given a location. So, it's actually going to be item based um collaborative filtering. So, we'll we'll lesson and kind of walk us through that example. But because again, it's the
[47:47] it's sort of like when you're browsing a product, you know, if on Amazon or something and you're on that product, we want to see other relevant products that recommend. Same thing here. given a tourist location, what are other tourist
[48:02] locations that that are recommended? Um, and so we want to follow the itembbased collaborative filtering approach. Um, because it's it's not like given a user,
[48:14] what are other what are recommendations we can make for items given that user um it's it's given the item in this case, right? Given the the location, then we recommend other locations. So we'll do item based uh collaborative
[48:30] filtering. So we'll go back and review that today. Um so that'll be super that today. Um so that'll be super relevant for this um project is is that recommendation uh example will follow pretty much identical to that you know
[48:46] pretty much identical to that you know given the ratings data um given the ratings data that we have and the the place ids and user ids.
[48:59] All right. So just to recap that um this second project has two parts. It has a image recognition transfer learning part with those uh structures images and then
[49:11] it has this recommendation a little bit of data analysis but recommendation uh of data analysis but recommendation uh system part um too. So those main two those topics uh this would be a great project to to do. Um and so the relevant
[49:27] resources for you if you really want to do this one um are going to be the do this one um are going to be the transfer learning lesson in deep learning uh which we will go back and take a look at today and then
[49:39] recommendation systems particularly item based collaborative filtering which we'll go back and look at too. So we'll do those two today. third project here and then after we do that we'll start getting into some review go back through
[49:53] some notebooks and some examples we did um you know that are going to be I think relevant to these projects. Um so the third one is uh um has to do with sales
[50:06] third one is uh um has to do with sales forecasting. So um in this situation we forecasting. So um in this situation we are um looking at uh using a model to are um looking at uh using a model to forecast sales and demand. Um
[50:21] uh and so this is going to be an interesting kind of regression oriented uh uh project that is dealing with a bunch of sales data. So uh if you're
[50:33] interested in these kind of problems um this might be a good one for you. Um and this might be a good one for you. Um and so uh we'll talk about it. Um so we want to predict the demand for various uh items across restaurants. So the sales
[50:47] that we're dealing with or sales data that we're dealing with are for various items that are being sold in restaurants. Um and so we have a bunch of data around that and we want to use that to kind of
[51:00] that and we want to use that to kind of forecast um the sales figures for forecast um the sales figures for um different time periods essentially um rel and and of course this is all related to uh the v uh various data sets
[51:15] that we'll have around um these products and how much they were sold over uh a period of time. So we'll talk about that. Um so you're going to be working with a few data sets in this. Um namely the uh first one is
[51:33] just some information about the restaurant. So uh each one has a restaurant. So uh each one has a restaurant ID and uh a name. So very very simple um as you can imagine um just basically an identifier and then
[51:47] some name that goes with it. Um so these are the various restaurants. Um now this data is probably the most important is the sales CSV. So uh we have different
[51:59] items that were sold um for different prices and how many counts of them and which items they were. So there's every row has a date of the transaction row has a date of the transaction um what item it was, what its price was,
[52:14] um what item it was, what its price was, and then um the which is a a unit price and then um the which is a a unit price per item. and then a uh total number of per item. and then a uh total number of those items that were sold. Um so this
[52:27] those items that were sold. Um so this is the sales uh data and then we have uh information about those items. So we have a store ID which corresponds to the have a store ID which corresponds to the restaurants. So we can figure out those
[52:41] restaurants. So we can figure out those item uh and and we have an item ID. Um and we it has a name, it has a calories, and it has a price um which is the same price that was in the sales uh CSV as well. Um [clears throat]
[52:57] well. Um [clears throat] uh so we have um uh items here. So we have three different data sets going on. And as you can imagine, it's going to be And as you can imagine, it's going to be useful to combine this information. Um,
[53:12] so in this project, you'll get some practice merging together these data sets. Um, for instance, like here we have restaurant ID. So of course, we can merge that to this and basically have a restaurant name per row. That should be
[53:25] restaurant name per row. That should be doable by joining those together. Um and then we should be able to figure out the sales as well by merging in that sales CSC because we have um you know uh product ids or item ids per sales
[53:41] record. So we should be able to join that to this data as well. Um so we know the name and the calories and the store uh for every single item that was sold. So all three we should be able to join all three and that's going to be one of
[53:57] all three and that's going to be one of the tasks uh in in in the task list. Um so just breaking that down the restaurants obviously as we said only has those two columns which are the ID and the name which is a string uh name
[54:10] and the name which is a string uh name for the restaurant. Um then we have the items which has an item ID, a store ID, the name, um then the unit price that we talked about uh earlier, the name of it, that's another string
[54:27] kind of fold string description of what the the name of the item is. And then we have the sales. Um, and then we have which has the data we really care about, you know, the the date. Um, and then the
[54:41] you know, the the date. Um, and then the the item ID, the price, and then the, the item ID, the price, and then the, uh, item count. Um, and so, uh, this is all of this data is what we're going to use to do some analysis
[54:54] and then ultimately building some, uh, forecasting models off of, uh, to to forecasting models off of, uh, to to predict the sales.
[55:07] So getting into some of the tasks here. Um the first one is uh uh um the first one is uh importing the data set into Python. So obviously that's just loading
[55:19] Python. So obviously that's just loading the CSVs um and then doing some uh info and head. So doing the usual just basic checks. Um and then looking for outliers
[55:31] checks. Um and then looking for outliers in any of them. Um so uh probably the one that we'll care about the most is the sales CSV. See uh seeing if there's the sales CSV. See uh seeing if there's any um outliers there. Um
[55:46] and so again that's up to you to choose what methodology you want to use for outlier detection. Um you can use the zcore method. You could use something more advanced like an isolation forest model from from scikitlearn. Um so but
[56:01] look for outliers. Um now this is the big part is merging. So we want to take those three that we have and merge them together. And so have and merge them together. And so what we should have is a full data set
[56:16] what we should have is a full data set that has all of these columns included in it. Right? So it should have uh date, item ID, price, item count, item names, item ID, price, item count, item names, calories, store, store name. All of that
[56:30] information is being brought together in a single data set. So the the way I recommend doing this, by the way, is in two steps. So I recommend merging these in two steps. Um, so you first want to merge, for example,
[56:47] so you first want to merge, for example, you could merge the restaurants data um to the items data. That's that you could do it that way. And that gives you a new data frame. Then you take that resulting data frame and merge it with the uh then
[57:03] data frame and merge it with the uh then merge it on the sales uh data frame. Um, and so, uh, you want to do that in two steps because remember when you do the PDM merge, um, you really can only, uh, merge two
[57:18] at a time. So, you you pick your two data frames and you pick the column that you're using as the one to look for the overlap. Um, and so what you likely want to do is merge uh, pick two of them to merge and then and then take that result
[57:31] and merge it with the third one. And so by the end of that you should have a single data set of everything merged together. So for example like um you can merge together this restaurants um with the items. That's a really easy merge
[57:46] the items. That's a really easy merge because this is the ID of the store. So we should be able to merge those together on this store ID. So that together on this store ID. So that should be not too hard to do. um because
[57:58] their restaurants CSV has everything um you know with that store ID. Um so by the time we merge these two data sets, what we should have is an extra column
[58:11] that represents the name, right? We should have a restaurant name that's merged in here. So we should have that added to it. And that's just from the merging of the restaurant CSV to this one. Then we take that result we take
[58:25] that result data frame and merge it to the sales. So we should be merging it to the sales and the ID we can the column we can use there to look for overlap is obviously the item. We could use the item ID
[58:39] because every row in the sales has an item ID that's being sold, right? So we can merge that here. And what that should do is bring in extra columns that should do is bring in extra columns that are the um the name of the the the name
[58:55] um of the or sorry the sales like amount like the item count. It should bring in like the item count. It should bring in the uh price um and it should bring in the date which is important to have too. Um so
[59:09] we should be able to merge those three together um and you end up with a final data set. So that'll be, you know, a little bit of work there to bring all those together. But that's the point of that task, merging all of it together.
[59:23] Um, and we should end up with all of those columns. Okay? And so if you want to if you want a good resource for doing that, go back a good resource for doing that, go back to the uh the data science course that
[59:35] that um lesson 9 notebook that we walked through on Monday that has those examples of merging, right? that those are really good examples for doing this kind of merge here. Um, so you definitely want to look at that uh as a
[59:48] good reference. That's in that's in the lesson 9 notebook in the data science lesson 9 notebook in the data science course. Okay. Then uh once you have all that data together, we then want to do some
[01:00:02] data together, we then want to do some exploration of it. So, um, one thing we could do is, so when it says examine the overall sales, um, uh, I think what a good place to start there is is, um, plot this based on the
[01:00:18] there is is, um, plot this based on the date. So, um, essentially what you need date. So, um, essentially what you need to do is generate a new column that is to do is generate a new column that is basically the the total sales. Um, which
[01:00:31] basically the the total sales. Um, which should be easy to do because um, you should be easy to do because um, you know, you just need to multiply together uh, you just need to multiply together the um, price times the item count cuz
[01:00:44] this is going to be a unit price that we're selling it for. And then we have how many of those items we actually sold. So if you multiply these two sold. So if you multiply these two together, you get a new column that is
[01:00:56] like the sales for that date and that item, item, right? And so the the point is uh once right? And so the the point is uh once we do that um we could group by the date
[01:01:09] and total this sales. We can sum the sales column grouping by the date and we want to generate basically a plot that shows the sales per day, right? I think that's that's ultimately like the best visualization you can get. But you have
[01:01:24] to do a little bit of work there to um group by this date and and sum this uh sales column, which I recommend generating. So the sales column is easy. You just create it by multiplying these two together. Um and then you can group
[01:01:42] by the date and sum this sales column per day. um and then look at the sales per day. Another thing you can do is once you have this data frame grouped by once you have this data frame grouped by day, you can reample it. Remember how we
[01:01:57] day, you can reample it. Remember how we had the uh we had that reample function. had the uh we had that reample function. Um so you can reample it to uh basically go from daily data to weekly and monthly. So um another so you can look
[01:02:13] into the reample function which um remember you can do like reample and you remember you can do like reample and you can pass in a like a w here or a month can pass in a like a w here or a month here capital M which will um basically
[01:02:28] aggregate it up to weekly data and monthly and then you can plot that and see how that changes. So all of these will be valuable to do for that exploration. These are things you'd want to do. We
[01:02:41] These are things you'd want to do. We have some examples of this in our uh um that and look through some of those notebooks we did like in lesson four with the data frames um we have some examples of doing resampling
[01:02:57] uh which remember just aggregates based on the that date uh uh column on the that date uh uh column essentially.
[01:03:09] Okay. So uh that's that's what these first few um that's what these first few you can answer all of these questions by doing different graphs like that. So all of these are really uh um
[01:03:24] uh answering those questions. Same thing with this like you can look at different with this like you can look at different quarters. So you can reample to the you can do reample um and pass in Q which will be a
[01:03:36] quarterly reample um from those daily sales. Um so again the key here is generating that daily sales column and then just resampling it
[01:03:48] um at different uh levels like monthly, quarterly, daily um or weekly. Um so those will answer kind of all these here parts A, B, C and D um can all be
[01:04:01] answered from that that kind of view. Um then finally and on this page what we of different restaurants. So what we can do is group by restaurant. So group by
[01:04:14] do is group by restaurant. So group by restaurant ID um and figure out uh we can group by restaurant and sum the sales to figure out what the top and then sort it right. So we can figure out what the top sales selling restaurants
[01:04:26] were. Um, and we can even look at that across different time windows. So we can group by uh if we produce this quarterly reample or or I should say uh monthly reample, yearly reample, um we could group that by restaurant ID and figure
[01:04:43] out who had the most yearly sales or who had the most monthly sales. Um, so this part E is going to be grouping by the restaurant ID.
[01:04:55] So, um, you know, that shouldn't be too bad once you've done all this work. Uh, grouping by the restaurant should be relatively easy.
[01:05:09] let me pause there. Um, any questions on this analysis so far? done a little bit of this. We talked about it yesterday.
[01:05:22] I say yesterday, I meant Monday. We talked about a little bit of these data some more of that, especially the merging here in this one. Um group by
[01:05:34] uh we sample. The key though is going to be generating that sales column and then grouping by the date to generate a new data frame the date to generate a new data frame that has daily total sales.
[01:06:07] couple more is we can also group by the item. So we can uh group by the item to figure out uh and we can group by item and store. Um so find out where the which item was the most popular at each store. That's that's just a simple like
[01:06:23] group by two columns the the item ID and the store ID. Um so so that'd be part F. Um, and we can figure out if the store with and we can figure out if the store with the highest volume, meaning so volume is
[01:06:38] the highest volume, meaning so volume is uh item count essentially is also making the most money. So, do they have the most sales per day? Um, that's an easy calculation to do. Um, once you have that full sales data frame and you can
[01:06:53] aggregate like the item counts too, um, and and aggregate the sales uh, as well. So you may have to go back and do that for part G is just um when you do the sum you group by the date you can sum the item count and you can sum the uh
[01:07:09] that sales column to figure out uh you know um what the volume is and the money um and you can average that out per day. Uh
[01:07:21] so and then you can look at the max. So you can look at the most expensive. So you can group by the restaurant and take the maximum uh item price and so you can figure out what the most expensive item at every restaurant is and and you can
[01:07:36] also look at what that item's calorie count is. So this is just breaking down, you know, we have this data. we can group by the restaurant and look at the max price. What is the most expensive item? So that's just an aggregation on
[01:07:51] item? So that's just an aggregation on the price. Um and you know that item will come with a calorie count because every item does. So we can grab that Okay. So bunch of group buys that you're going
[01:08:05] to be doing. So in this project uh in the analysis you should find yourself doing group by a few times uh in here. Um and then and then we pivot to part two. So that's part one is just analysis. Um part two is building the
[01:08:22] model for forecasting. So part one just analysis. Part two is building the model. Um so and you're actually going to be building a few different models to do regression.
[01:08:36] So, uh, you're going to be using a linear a basic linear regression, a random forest regression, and an xg boost regression. So, three different boost regression. So, three different regression models. Um, so, and um, it
[01:08:49] even has some suggestions for different features um, like what the quarter is, the month, the day of the week. Now, remember, you should be able to generate these features from the date. So, um, this is
[01:09:04] features from the date. So, um, this is an excellent practice of changing over an excellent practice of changing over the date into a datetime object and then the quarter, grabbing the month, grabbing the year,
[01:09:17] day of the month. Um, those are all things that come from the datetime things that come from the datetime object. So, uh, you you should be able to extract those, um, from the datetime object. And so
[01:09:30] really recommend mapping that date to a datetime object which we saw how to do on on Monday. That's not too hard to do, right? It's just using that PD2 datetime right? It's just using that PD2 datetime function to do that. Um and so then you
[01:09:45] want to split apart your data into training and test. The the big thing training and test. The the big thing with this data that is indexed by date is keeping it in order. So you really want to sort by the date. um and use the
[01:10:00] want to sort by the date. um and use the last 6 months as the test and and the um previous data to the last six months as the training data. the training data. Okay. So this will involve sorting
[01:10:14] sort by the date column so that you have sort by the date column so that you have it all in time order and then use the um use the first section of the data. Basically everything but the last 6
[01:10:27] months is going to be the training data. The last 6 months worth is going to be the testing data. So I'm going to sort by that date. Um now you may be wondering what am I predicting? So the thing that you should
[01:10:40] be trying to predict are the total sales. So it's that it's that sales column that we generated from the um item count times the price is the sales quantity. We want to predict that. So, it's the sales amount. Um, and we want
[01:10:58] to build a regression model to predict that. Um, and and remember, we're going to have a total of that per day essentially, right? Um, and so we want
[01:11:10] essentially, right? Um, and so we want to uh be able to forecast that uh um at different time intervals. So um essentially what you're going to do is build you're going to build a model using all those features to
[01:11:25] predict what the sales uh quantity should be. Um should be. Um and uh you will um use RMSSE. Now that's and uh you will um use RMSSE. Now that's a very natural uh evaluation for the
[01:11:39] model um because we're doing a regression, right? MSE we've talked about that many many times. um that's what we would want to use to uh you know if we're doing um a regression problem. So that makes sense. Um and then you so
[01:11:54] you'll compare the performance of those three. So the linear regression, random forest, XG boost, you'll train all three, take the one with the lowest MSE that's going to be the best model and then use that to make a um prediction
[01:12:11] then use that to make a um prediction for the next year. So um you'll forecast you'll use that to make a forecast for the uh next year which would mean um predicting essentially making a prediction um at the year level. Um so
[01:12:26] what would be kind of a what would be the next year's sales quantity?
[01:12:41] Any questions on this? here and we can do that today is going back and looking at the resampling. We
[01:12:53] can do that uh because I think that'll be helpful for this. It we had examples of how to do that I think early on in our uh work with data frames. We can certainly go back and and look look at the resampling examples because I think
[01:13:07] that'll come in handy. Um and also, you know, maybe group by operations too because this one's going to involve a lot of uh group by operations which I think are uh really handy. Um you know, we've looked at them before, but it's
[01:13:22] obviously been a while since we've had to do those. So maybe we can go back and to do those. So maybe we can go back and look at those two for review. Okay. So this one just to recap has two parts.
[01:13:35] Just exploring the sales data which is a lot of uh like I said group by different manipulations maybe even building some graphs um and then uh building a model
[01:13:47] off of that sales data to do to predict um the sales quantity. Um and so if you you know as I said this might be a good project if you enjoy uh building those kind of models. Um so th
[01:14:02] this one the good thing about this project is it doesn't involve any deep project is it doesn't involve any deep learning. Um but it is using a lot of older content from you know doing uh regression and doing a lot of group by
[01:14:16] and resampling and such. Um so so you have one so you have an option to do any one of these three right. Um so
[01:14:28] do any one of these three right. Um so uh really good options I think. So those are the three that we have. Now what I want to do is spend some time uh
[01:14:40] starting now reviewing some of those topics that I think are going to be topics that I think are going to be really relevant. Um the first place that I'm going to start is uh going back to the first project on the autopilot in
[01:14:53] the first project on the autopilot in Tesla. Um I'm going to review the object detection uh model that we worked on uh not too long ago. You know, that was not too long ago. You know, that was recent in our deep learning journey. Um
[01:15:06] I'm going to go back and take a look at that. Uh so let me pull that up. um that would be within I think lesson nine of our deep learning course.
[01:15:18] our deep learning course. So let me pull that up. Sorry, lesson nine was transfer learning. It's lesson 10.
[01:15:48] project is going to be a really fantastic resource uh object detection with YOLO. We did this a while ago. Uh but I I do would just want to briefly walk back through it as a you know really good resource for that first
[01:16:02] really good resource for that first project because obviously um in the in the beginning of it the first part of it is do is building an object detection model. Um so the the idea there is you should be able to follow
[01:16:16] this notebook mostly all the way through. It's just you're going to be working with different data obviously with the uh um Tesla vehicle um data set. Um so let's go back through this. So
[01:16:30] so let's go back through this. So remember we had to do uh um some work remember we had to do uh um some work here with the YOLO um repository and we use some of those built-in scripts to do the training and the evaluation.
[01:16:45] the training and the evaluation. So uh um and we had some helper So uh um and we had some helper functions along the way. So we had this functions along the way. So we had this function to uh download a bunch of data.
[01:16:57] Now will we need to to do that for this uh for the project? No, because you already have the data. Um so we won't need to go out and and download it. So you won't need a function like this necessarily which was used for
[01:17:11] retrieving data from the internet. Um so we didn't really need to do that which was pulling from this you know public data set you will already have um data
[01:17:23] but what's nice is like if you want to use it in this format you or with this um kind of setup you'll want to make sure that the data from the data set is
[01:17:35] in this kind of uh format. So, you want to have it in an images. You want an images folder. You'll want a labels folder that has these uh text files. And remember, these text files are unique in that they have the the box
[01:17:51] coordinates on each on each line along with the class of what vehicle it is. Um, so we want uh all of that to be now that should already be set up for you, but you just want to double check it's kind of in this uh this kind of folder
[01:18:05] setup where you have an overall folder with an images subfolder, a label subfolder, and they have the images are within this one, and then the label files are within this one. We won't be downloading it from a location. We'll
[01:18:18] already have it. That that's just the difference there, right? Um, so you'll you'll already have it from the data sets. Okay. sets. Okay. Otherwise, we cloned the model. So, um
[01:18:31] we cloned the model from this repository uh because it unfortunately it's not one of those that are built into caris or built into scikitlearn or something like that. So, we're pulling it from this repository so we can use it. So, we did
[01:18:45] repository so we can use it. So, we did that. We set up a results directory to that. We set up a results directory to save our training um artifacts. So that's something that we still would use in this setup. Um, you know, this is
[01:18:58] just setting up a folder to save the results to um for training. Um, and then essentially what we're leaning on here is this training script.
[01:19:12] So this is using this is a Python script that is using uh um you know a training loop. And remember when we looked at PyTorch, usually that's a very manually
[01:19:27] coded out loop, right? It's because in PyTorch, your training is a little bit more explicit, right? It's not just model.fit, which we see in TensorFlow. Um, but this YOLO model is a PyTorch model. So the code to do that training
[01:19:41] is within this training.py script. Um, and we're going to run that here um using the existing model, but we're going to train it against our uh um data
[01:19:55] that's set up that's uh in the in the right folder right in the um it's assumed to be in this setup that we talked about earlier uh with this directory. Um and [clears throat] then we will uh execute this. So this what
[01:20:12] following this example is you want to make sure this part runs um when you download uh this when you clone this repository
[01:20:24] um you may if it's not working for you you may need to tweak this data.yamel to point to the right folder where you have your data set up in on on your machine. So, if you're doing it locally, um, or if you're doing it in Collab, um, you
[01:20:40] just want to make sure that you have your data set up. Uh, want to make sure you have this. This is just a configuration to say where on my machine is the data stored. Um, so you just want to look in that file and make sure it's
[01:20:53] pointing at the right location so it can pick up your your data set. Um, and you know, you can also play around with how many epics it's running. So, and and what I would recommend too is using a GPU, especially if you're on Collab,
[01:21:08] connect to the GPU, and this um should speed it up ideally, right? It should speed it up. Okay. Then we had a helper function to Okay. Then we had a helper function to visualize some of our results. Um so,
[01:21:23] this is this should be exactly how it is. It's going to look inside of our um is. It's going to look inside of our um path and pull in some images from our uh path and pull in some images from our uh training um run and uh um then we can
[01:21:39] visualize what some of our results will be. This is a helper function to do that be. This is a helper function to do that um from our results directory. Data.l will be no what I'm saying is in this is a file that has that. So, uh,
[01:21:53] let me run this actually to show you what I mean. Um, it may help to just run through this real quick. And I don't need to be connected to just
[01:22:05] connect to CPU to do this. to do this. So, data.l is a configuration file that should have a default directory. What I'm saying is if if this uh training
[01:22:19] isn't working for you, you may need to change some of the configuration change some of the configuration settings in that YAML file. Uh so I want to see what it actually contains. Um what it's pointing at initially.
[01:22:35] Um what it's pointing at initially. So let me go down and just clone this. it here it's just pointing to so it has some meta data about the data set. So it
[01:22:51] has the this is what I'm saying is you may need to change this location. This training data which is inside of a folder um that is called that is uh set
[01:23:03] up to be so dot dot means go back a directory and it's inside of a folder called train and then there's a folder within that called images. Um so this might change depending on like if you're running this locally on your
[01:23:18] own machine this might be a different directory. It depends on where you're directory. It depends on where you're saving your data set. Um but uh what I would recommend is trying to follow this example as closely as possible which
[01:23:32] example as closely as possible which means that um you so notice how the yolo means that um you so notice how the yolo uh the the um the the data is within a train and validation folder that is underneath the YOLO uh directory. So
[01:23:49] once you clone the yellow directory, you can copy all the data in there in these folders which should line up to this exact configuration. You shouldn't need exact configuration. You shouldn't need to change any of this. Um notice like
[01:24:01] to change any of this. Um notice like now these might change as well. Um in fact these are likely to change which are going to be your uh labels for the are going to be your uh labels for the different vehicle types. So, um, you
[01:24:14] know, these are going to be, uh, different vehicle types that, um, and you should be able to get that from the, uh, the the label files, um, themselves.
[01:24:26] You should be able to see what the the classes are for the various images. classes are for the various images. [snorts]
[01:24:39] It should be in the data set uh, description too, I think. this, it should mostly follow this example, we have a helper function to do
[01:24:55] inference. So this is to make predictions, which should mostly lean on running this detect.py script, which is, you know, the script that basically does forward propagation for us with the PyTorch model. um using our training uh
[01:25:09] results to load in the model. Um and then it's going to store the prediction results inside of this directory. And remember the prediction results are um interesting for object detection, right? They are the box plus
[01:25:25] the label of what's inside the box, right? And so for a single image, we may have multiple boxes. That's totally possible. Um, so the the predictions are not just a label, but boxes with labels, right?
[01:25:39] And here's a helper function to visualize some of the results. And again, this is something we uh want to do and and the project even asks you to do is to visualize some of the uh detections. Um,
[01:25:54] and so this is going to uh run some inference and ultimately uh visualize inference and ultimately uh visualize it. Uh and so we can see what some of it. Uh and so we can see what some of the images are with those boxes. Um so
[01:26:07] we this is some examples we saw uh earlier. Um but as long as you mostly follow this example um you should be good should follow mostly this. The only
[01:26:21] difference really is instead of downloading this public data set you're going to be using the one that is given to you with the project, right? And so you just want to make sure that is set up in the right place um for the
[01:26:35] training to work and the predictions to work example. Okay.
[01:26:47] So this is a great resource this 10.07 notebook. If you plan on doing that first project this is a very very good resource for you uh to work with there.
[01:27:21] to, um, let me go over to some transfer learning practice, which I think will help with the second project. Um, so
[01:27:34] inside of lesson 9, we did transfer learning and we had a notebook there to learning and we had a notebook there to practice um to practice doing transfer learning. So, let me share my screen and show you that one. So this will be
[01:27:48] really really helpful for the uh second project. So this is within um lesson project. So this is within um lesson nine of uh the deep learning course and this was the the only notebook in there 9.04. This will be a really fantastic
[01:28:03] example to follow for that second project. Remember what you're doing there is predicting the um various structure images and categorizing them into one of those structures like the the statue or the um
[01:28:19] stained glass or whatever it is, right? The different um tourism structures. So, uh, the reason this is so relevant is because, um, it's going to walk you
[01:28:31] through, um, using a model off the shelf, which I highly recommend for that second project, and then just, you know, training it against that data that you have, um, with the which are going to be those images. Um, and so, uh, this
[01:28:47] example you uses VG16, which you're welcome to use as well. Um so here we welcome to use as well. Um so here we import this uh in order to use it. Um,
[01:28:59] and remember this was the key was uh this is what makes everything work is loading in that model here but making sure we turn off that top of the network um by by making sure we do include top equals false.
[01:29:15] um which you know will uh basically leave out the end of the network so that it allows us to put in our own layers there to do the classification. So essentially this is going to be our starting point.
[01:29:31] this is going to be our starting point. And then we um take this and we go ahead and flatten this result um from that last layer of the VG16. And then we put in a dense here with a relu. And then we put in a dense here
[01:29:45] relu. And then we put in a dense here with a softmax. And again this number of with a softmax. And again this number of neurons here should match to how many categories we have for that project. Right? So um if I go back to that
[01:30:00] project um in part one uh there were these categories right alter apps bell tower column dome two different types of domes flying buttress gargoyle stained glass vault. So I should have as many you know
[01:30:17] um categories there as I as many neurons in that last layer as I have categories here uh which are in these different folders right um so I this this we want that to match that number um as far as our output and
[01:30:34] one of the things that it wanted us to do as well was add in a dropout layer um do as well was add in a dropout layer um so that would be worthwhile doing is you putting that in here. And we could, you know, we could drop out uh right here um
[01:30:48] after this um to affect some of these neurons being turned off. Um so that's something we definitely could do. Um but once we do that, we see we have
[01:31:01] all of the VG16 layers, you know, populated in here, which are all these blocks of convolutions and max pooling and convolution, max pooling, and all that. And then basically here's our final layers we added in.
[01:31:16] And then we want to train this on the uh then we want to train this on the data then we want to train this on the data right itself. So um this we would want right itself. So um this we would want to do these kind of steps uh to
[01:31:29] uh to then go ahead and set up our training and go ahead and set up our uh training and go ahead and set up our uh uh model.fit and and all of that. uh model.fit and and all of that. So um you know that would be something
[01:31:44] uh worthwhile doing is basically looking at this example as a good way of setting our network up and then what we could do is look at other examples that we had for um you know doing the training process especially with uh image data.
[01:32:01] So that may be something worthwhile to uh look at and I can do that. Um, let me find well before I find a new another example, especially with setting up the data loading, which I think will be relevant. Um, any questions on this?
[01:32:18] So, so like this is what you'd want to be doing for this second project, that part one, right? Is using transfer learning to your advantage? And really for you, right? It's just building your
[01:32:30] it ready for training by doing model.compile. And you still have to actually train it by doing model.fit. You're still doing that, but this is giving you your model architecture, right? Is you're pulling
[01:32:43] this off the shelf and then adding in a few layers at the end to do the We practice that a little bit. This is a great resource for setting that up. Um,
[01:33:00] great resource for the second project. Uh this is a good example that you could follow on using one off the shelf. We actually have multiple examples I think actually have multiple examples I think I think we have one later on uh um of of
[01:33:14] I think we have one later on uh um of of using transfer learning as well. have another example in mind that I wanted to use.
[01:33:36] the It's the roses. And remember, we loaded those like rose data sets so we could uh Yes, this one. um as a good just to show Yes, this one. um as a good just to show you like a good way of setting up the uh
[01:33:51] setting up the data here. So let me share my screen. It's this one.
[01:34:03] think this is a good resource for that second project is the um you should be second project is the um you should be following this the way this loads in the data because the data that you're going to have remember is set up in these
[01:34:18] folders. So I go back here. It's set up in these folders like this. Um and and in these folders like this. Um and and this uh will assume a similar setup. Um so this is a really good resource for examples on how to load in the data set
[01:34:33] for working with caris um and tensorflow. So remember we download the data set here here um and what we did is we uh basically
[01:34:47] um and what we did is we uh basically um used a function. If I keep going here by the way this is a helpful this is a helper function to to just display images. Um, so this displays an image given uh the directory.
[01:35:03] So that's that's a helpful example. This is what I really wanted to show off was is what I really wanted to show off was for that um transfer learning example when you're training. This is how you want to load in your data exactly like
[01:35:16] this. So you want to use this utility image data set from directory because our images are already organized into these folders right with the labels. Um these folders right with the labels. Um and this function assumes that structure
[01:35:31] and so we can pass in the directory path. Um and basically uh this will load this will read from those folders and organize them into their classes. Right?
[01:35:43] So you can see how many classes there are. Um, and you get this confirmation that it found all these files belonging to five different classes. And you can set up how much data you want to reserve for testing and it will split that out.
[01:35:58] Um, and so it reserves a certain amount of them for training and then it reserves the remainder of them to be used for validation. Um so really excellent uh reference here for how to load data from that kind of uh folder
[01:36:15] setup uh which I think you would want to do for building out your training set and validation set. And then you can visualize by the way you can also get the class names from the this class names attribute. So once
[01:36:28] you've loaded in from those folders, um you can grab the class names from here and you can see like you should have all those names that match this. You should have all those exact names that match your folders. Um and so
[01:36:42] that's how you can that's how you can verify that it loaded those correctly. This is a really good function to because you know in the project instructions it wants you to plot some of the example images. This is a really
[01:36:55] of the example images. This is a really good um example to grab a batch and go ahead and plot them. So it uses the plt imshow function to um plot and then it imshow function to um plot and then it puts a title on it with the label,
[01:37:10] right? So you um grab the label of the image and which is an index and you use that index into the class names list to grab the string. uh uh and so that puts it on the title and and so this is a really good way to
[01:37:26] kind of sample visualize some of these images uh in this subplot. So it plots the first nine images. Obviously you can do more than that. You could build out uh 20. You could do however many you want to do, but um nine is a nice number
[01:37:42] want to do, but um nine is a nice number because it's this 3x3 grid of images.
[01:37:55] Yeah. So in this situation um it's organized like this where there's a there's like a data set directory here and underneath of that are the folders with each uh type of flower.
[01:38:10] Yeah. So we had each one of those. Um, so when organized very similarly, right? It's going to be a folder that has a bunch of
[01:38:22] subfolders that represent each class. So images of altars, images of bell towers, images of columns, images of stained glass, right? It's going to be set up in the same exact setup. That's why I think this is a good uh very good example
[01:38:40] of how to load in a training set and a validation set given this directory setup. So when you specifically have that directory set up, this helper that directory set up, this helper function can uh grab that data for you.
[01:38:53] Yeah, which I recommend doing. So I think for that part one of the second project, I think you're going to want to lean on this as a good example of loading the data. Um, and not only loading the data, but training a model,
[01:39:07] right? So now obviously in this one, I think we built our own convolutional think we built our own convolutional net. Um, but this also, by the way, has a really, really fantastic example of adding in augmentations, which we did
[01:39:22] adding in augmentations, which we did not do on the uh, VGG16, but something we certainly could do. So, what the project wants you to do is set up training it with and without this added into the model. So, there's no reason we
[01:39:36] can't take the existing one and in the very front of it build a sequential that has this first with some flips, rotations, and zooms added in there. Um,
[01:39:49] and we can put this first and then have the rest of our model second um in the sequential, right? Because you you're allowed, remember in caris, you're allowed to put a sequential within a sequential. So you can so this is a good
[01:40:04] example of that where you put this augmentation first and then the rest of the architecture and now this is a custom model that we did ourselves custom model that we did ourselves but you know there's no reason we we
[01:40:19] here rather than having a very custom architecture like this. Um the rest of this model is basically going to be the the the one we already built with the BG16 and our custom layers at the end. the only thing we're really adding to it
[01:40:34] is this augmentation. This is a good example of that. Um and This is a good example of that. Um and then we uh we summarized it. We compiled then we uh we summarized it. We compiled it. So this should look very similar um
[01:40:49] uh using atom using accuracy using cross entropy. Now remember this one did not have a right? It doesn't. So it it does not have a softmax here. Um so therefore we
[01:41:05] had to turn this on to true. But the default behavior would be to have a softmax and then we can uh leave this out or just set it to false which is the default. Um so that's the only change from this example of the of the compile
[01:41:19] which um which if I go back to this we added you know here um you could just use this which by default assumes we have a softmax you can just use the string. So that's the only thing that would
[01:41:34] really change and then we do uh model.fit right model.fit fit. Uh we pass in our data set, we pass in our validation data, and we can change the epics, etc. This is another one where I would
[01:41:48] recommend using the GPU if you're in Collab. Um because when you're anytime you're doing these vision kind of tasks, right, I think using the uh using the right, I think using the uh using the GPU is going to be beneficial.
[01:42:03] GPU is going to be beneficial. Okay. we can always predict on it. So uh we can do model.predict on a batch of
[01:42:19] images too from our test set to kind of validate our predictions. Um so that's you know something else we could do. All right. So this combined with this
[01:42:31] should be enough to get you going on that second project on the first part. Um, I think those are really good examples uh that you would you would examples uh that you would you would want to follow um especially with the
[01:42:43] want to follow um especially with the augmentation and the uh the augmentation augmentation and the uh the augmentation and the um the the VG16 example in the 9.04 notebook. The only thing that we do not have here an example of is the uh
[01:42:58] early stopping. So if you want to add that to the mix uh you certainly can. So um remember that would be doing something like uh this where you you build a callback um which looks like this where you
[01:43:13] monitor some metric you do the TF caris callbacks early stopping and then inside of the fit you just add that call back right so callbacks is early stopping um
[01:43:26] so that's just a simple addition I forget exactly which notebooks we had this example in but you know we've done that before I think the project asks you to do this too is add in early stopping. Um,
[01:43:40] but that's really simple to add, right? Really, really simple is just setting up a early stopping callback and then making sure you put this in here in the model.fit inside of this callbacks list. Um, the other thing you could do, you're
[01:43:56] not required to, but maybe you know if you want to practice it, it may be worthwhile is setting up tensorboard. Right? We talked about um setting up a tensorboard callback um and then you can put that here in the callbacks and then
[01:44:10] put that here in the callbacks and then uh launching the TensorBoard uh um launching the TensorBoard uh extension inside of the notebook and and visualizing the training. You don't have to do that, but you know, if you're
[01:44:23] of epics, it's it's usually valuable to see that, which I think will be a good example for again that second project. you're going to be doing item based collaborative filtering. So we have a really good example of that here which
[01:44:36] really good example of that here which you uh would be following. was saying is uh I wanted to resume here by walking through
[01:44:50] um an itembbased collaborative filtering example example um which uh will be really helpful for that second project, right? Uh because that's the second part of it is is um
[01:45:05] building out an item based recom recommendation system. Uh so I'll walk through that example here. that's going to be a little bit here. that's going to be a little bit down in the 7.01 notebook um under this
[01:45:18] section called item based collaborative filtering. that and we'll walk through it together. Um so if you remember one of our uh
[01:45:30] goals here with the collaborative filtering is to essentially figure out what items are similar to another. And the way we do that is through their ratings uh profile essentially right like what items gathered similar ratings
[01:45:46] like what items gathered similar ratings from users uh which we will um calculate through this cosign similarity function. And if you remember we used that um and remember what that does just uh conceptually is it looks at the angle
[01:46:04] between the two ratings vectors any two ratings vectors in the matrix. Um we can ratings vectors in the matrix. Um we can look at the similarity between uh and so remember the visual idea of that is like if two ratings vectors are very similar
[01:46:20] that means they have a very small angle between them. Uh and ones that are exactly similar would be right on top of each other. So they basically would have um a cosine similarity of one meaning their angle
[01:46:33] would be very very small between them. Uh but if they're uh dissimilar, meaning they're they're much different ratings vectors, they're going to be um wide angled here and very far apart, which would drive their similarity lower. So
[01:46:50] the similarity metric is between 0 to one. And this very wide angle would be closer to a zero similarity. Um a very small angle, meaning that the vectors are very close to each other, would be a high similarity, so closer to one. Um,
[01:47:05] and so that's going to be our primary metric for determining which items are similar to each other, right? Is is through their ratings data. Um, through their ratings data. Um, and so we will uh see that. Hopefully
[01:47:19] that rings a bell. Um, when are when are the projects due? Uh when are when are the projects due? Uh it should be uh um ideally what you want to aim for is is what we've always done is like 2 weeks after the last uh lesson
[01:47:35] is like 2 weeks after the last uh lesson which will be on Monday. Um but uh realistically you have until the end of your program to submit it in in practice
[01:47:47] because uh that's how long it'll be open for. So, um, which you should be able to find that date, you know, in your LMS, like how how, um, how much time you have left in your course. Uh, but you want to try to do it in a reasonable amount of
[01:48:03] time, right? Um, and I think these projects will be very, very doable, good examples to follow. I don't think they'll take that long, but yeah, a good goal is like two weeks uh, after Monday, two weeks from this
[01:48:18] uh, after Monday, two weeks from this upcoming Monday.
[01:48:31] example just to see it again. Um there's a fake data set uh with um items and users giving ratings to various items. uh obviously in the project you have a
[01:48:45] uh obviously in the project you have a real data set that has uh users and and items and ratings uh in a CSV right so you have the user ID you have the item or the in that case the item is a location right you have a place ID and
[01:49:00] you have the various ratings of those places um but we have a fake kind of a places um but we have a fake kind of a fake data set here um let's run this so Um, but you have something very similar to
[01:49:16] this uh that you can load into a dataf frame from the CSV uh um you know that you start out with in the project. So we have this dictionary um and then we can load that into a data
[01:49:30] frame. And so what we have is this data frame And so what we have is this data frame here. Um Yeah. So if you have any uh so if you have that data you'll basically load it
[01:49:46] into a dataf frame. So this is this is the kind of data that we have um in this the kind of data that we have um in this user item setup right so the columns are user item setup right so the columns are the users and the items are on the rows.
[01:50:00] This is the setup that you want to have. Now when you load your data into the Now when you load your data into the data frame from the data set it, you know won't look like this necessarily. You're actually going to have various
[01:50:14] You're actually going to have various columns. And so what you need to do to get this kind of matrix is you need to pivot it, right? You need to pivot that data set. There's actually an example of doing that I think earlier. Uh if I
[01:50:30] scroll up here um on the anime data set for this uh userbased collaborative filtering I just want to see um yeah so this is the example you this is
[01:50:45] the one you would want to follow uh to to kind of pivot that initial ratings um CSV uh you want to pivot it um now you what
[01:50:57] you actually want is the transpose of this because you you in the um item based collaborative filtering you actually want your rows to be your item. actually want your rows to be your item. So you do want to use the pivot table to
[01:51:12] generate your matrix, your data frame, but you want your place ids to be on but you want your place ids to be on your index and you want your user ids to be in the uh columns and then the values are the ratings. And so you should end
[01:51:27] up with a basically a matrix that looks like this but transposed, right? Your places are going to be here on your rows and your users are going to be here in the columns. That's what you want, which is exactly
[01:51:39] this setup in the item based. So notice the item based has this data frame that has the places or the products, the items are on the column or sorry on the
[01:51:51] items are on the column or sorry on the rows um and the uh users are here on the uh columns. And so when you do that pivot table, you're going to have null values naturally because not every user has
[01:52:05] given a rating to every place. So you're definitely going to have null values recommendation. Um and that's okay. You can leave it Um and that's okay. You can leave it null um because uh you know that's fine.
[01:52:20] Usually when you build the recommendation system, you're going to end up um filtering those out anyway by looking at the places that they've actually both all the users have given ratings to um and then using that vector
[01:52:33] to do the similarity. So when you're doing your example, you're going to load in your data. You want to pivot table it so it ends up in a setup like this. So just something to keep in mind for that project, right?
[01:52:47] keep in mind for that project, right? Pivot table to keep it like that. Pivot table to keep it like that. Okay. So, uh then we have a helper function to grab all the un uh unique items.
[01:53:00] Um uh that's just so we know what all the unique place ids are. This just a helper. Um so we have that list of unique uh unique place IDs um uh just
[01:53:13] for reference. So we can have those. So these are all our unique items uh from the data set. So we should be able to grab those. grab those. Okay. Now here is a really critical
[01:53:25] function that we would want to build which calculates the similarity between any two items. Now that's going to be really important because remember what really important because remember what we're going to do is essentially um you
[01:53:39] know given any item we're going to go through and try out every other item here and figure out what are the most similar ones. So, we'll always have one to start with, which is our given place, our given tourism location, and then
[01:53:54] we'll basically cycle through every other potential location and generate a other potential location and generate a similarity. Um, and and filter out to the top 10, top five, whatever for our recommendation. That's going to be the
[01:54:07] strategy for the recommendation, right? Um but in order to generate that similarity remember we have this function and one of the things that we do is this initial filtering to um
[01:54:22] given these two items that we're checking their similarity between is to checking their similarity between is to uh um go through and find ones that both have ratings. Right? Right. So, we want to make sure both of those have uh
[01:54:38] ratings across our users. This just eliminates the uh this eliminates the um the the places where one of them doesn't have a rating because then we won't be
[01:54:50] able to compute the similarity. So, essentially this iterates through every user and checks um did the user rate both items. So if they did, we add both of those ratings to uh this um dictionary uh so that we have every
[01:55:07] instance where a user rated both of those items, right? Every single row uh those items, right? Every single row uh where uh both of those were rated, right? So that's that's all we're doing here. Um and then essentially what we're
[01:55:23] doing is uh if there were no overlaps meaning there were no cases where both of those items are rated um we basically return zero meaning there's absolutely no similarity we can compute no similarity um that's uh basically we
[01:55:39] have nothing to compare because there were no instances where where two users gave ratings to both. um that's probably unlikely to happen, but in the event that it does, we just return a default value of zero similarity. Right? So
[01:55:53] that's just this case. Very rare case, but it could happen. Now, essentially what we want to do is grab all of those ratings that were um
[01:56:05] from uh you know item one and item two by the common users. So users that gave ratings to both. Um, and so we collect all of item one ratings and all of item
[01:56:18] two ratings across those users that rated both. And so we end up with these two vectors, item one ratings and item two ratings. And we compute their similarity through the cosine similarity function
[01:56:32] um of those ratings vectors. So um that gives us the similarity which is a value between 0 and one of these two items. So remember the logic there given two items let's look at all the places where users
[01:56:47] rated both collect all those ratings and compare those two ratings profiles to see how similar they are right okay so that's this function so essentially
[01:57:02] you would be using this exact function to compute item similarity um so pretty much that and we can test it here by looking at the similarity it here by looking at the similarity between the Dexter uh show and the Mr.
[01:57:16] Robot show. And again like in the project the items are places so they we could use a place ID so place ID 1 and place ID 10 which are two different
[01:57:28] tourist locations. Um we can compute the similarity. really critical in actually doing the recommendation. So this function here is
[01:57:44] recommendation. So this function here is um basically what will give us our top n um basically what will give us our top n items however many we want to do um in items however many we want to do um in terms of our uh our similarity. So
[01:57:57] essentially what we're going to do here is um given an initial item so that's called the target item. So given that initial place which is which is exactly
[01:58:09] what we want to do in the recommendation system is given a place to start at um system is given a place to start at um let's go through and for all the unique let's go through and for all the unique places let's iterate through and compute
[01:58:25] the similarity across every unique place that is not our target. So this is looking at every other place. How similar is it to our starting item, our
[01:58:37] starting place, right? It's exactly what this code is doing. And it's generating this scores list which is um examining that similarity score between every other target item and our starting uh target item.
[01:58:54] And so what we can do with that is um obviously we can sort this according to our greatest to least similarity. Um and so uh that's what this does. It Um and so uh that's what this does. It sorts it in descending order. Um and
[01:59:09] sorts it in descending order. Um and what we can do is um and I would recommend doing this is we can also put in a parameter K which could be um you in a parameter K which could be um you know defaulted to five for instance but
[01:59:23] essentially what we want to do is return the scores up to the the top the top K scores. So something like that which would return like the top five or if we
[01:59:35] change that to top 10, top three um that would be a more realistic kind of function, right? Is give us the top five, top three, not the entire list of that useful from a recommendations perspective. Usually, you know, the top
[01:59:51] perspective. Usually, you know, the top five, top 10 are usually perfect. Um so so that would we can put in a parameter K and we'll get only the top K scores uh there from that list which is what that would do.
[02:00:07] Okay. So then we can test that. Uh so given a starting item this gives us the similarity across all the other items. Uh and obviously we could um you know we
[02:00:19] Uh and obviously we could um you know we could uh uh pass in the parameter uh um top three which would um limit that to the top three. So if we run so we
[02:00:31] generate this function generate this function run that and then we run uh this
[02:00:47] and generate this. So this is only giving us the top three. It's not giving us the list on the entire every single similarity. It's just giving us the top three that we might recommend. given a starting place ID that's in our
[02:01:02] data, right? Given that, then we could recommend the top three similarities.
[02:01:17] this, this was also in the in this demo. The reason we don't need this is because we're not given a target user. If we were given a starting user, we could um
[02:01:29] were given a starting user, we could um filter to um essentially the ones that they haven't seen before uh and and recommend ones that they haven't seen. And uh but we're
[02:01:42] not that's not what we want to do in the project. The project is all about given a starting place, what are other places that we could recommend? And that's what this would do. So in the project, the adjustments that
[02:01:54] you need to make are you're mostly going to follow this exact example. Um, the are, you know, when you load that data from the CSV, you're going to want to
[02:02:07] use the pivot table, um, like what is used up above that I um, like what is used up above that I showed earlier, um, to initially load your data into the user item matrix, um, which is going to be that ratings
[02:02:21] data set. You want to load that one into a dataf frame, and you're going to want to pivot table it, but you're going to want these flip-flopped. You're gonna want the place ID as the index because you want that to be your rows and you're
[02:02:34] you want that to be your rows and you're going to want the uh the columns to be the user IDs, right? So, you're going to want this transposed. Um, and so you could like you could load it exactly like this and then transpose it. that
[02:02:48] would work too, but you might as well just save yourself, you know, that one extra operation by just ensuring that when you pivot table this um the index is your is the place ID, right? It's the it's the tourism location should be on
[02:03:02] it's the tourism location should be on the rows. to do that and then mostly uh follow up with uh the um
[02:03:17] you know with the uh item based collaborative filtering example. So, for that second part of of the second project. Really really good resource project. Really really good resource here.
[02:03:32] Any questions on this one?
[02:03:44] I know it's been a while since we did a recommendation, but uh hopefully that, you know, rings uh rings some bells for us and seems somewhat familiar. us and seems somewhat familiar. All coming back. Okay, great.
[02:03:58] Fantastic. Okay. So, uh, a couple more examples I wanted to run through that are going to be especially relevant for the third project. Um, so again, for the second one, the resources that are going to be
[02:04:13] one, the resources that are going to be helpful are the 9.04 9.04 notebook for helpful are the 9.04 9.04 notebook for transfer learning and the 8.07 071 for transfer learning and the 8.07 071 for uh um looking at the loading of the data
[02:04:27] and the augmentation part of the neural net training. Um and then this resources should really help you for that second project. For the third one, what I want to do is look at a couple of
[02:04:41] data frame operations that I think are going to be really beneficial. Um the first one is going to be in this um this is all the way back in our data science is all the way back in our data science course in lesson 10. So in lesson 10 on
[02:04:54] feature engineering um we had this 10.01 know one notebook which I think will be know one notebook which I think will be a good resource for um for for that a good resource for um for for that project uh the third one on sales
[02:05:07] forecasting just because it has some good examples on grouping right and I think you'll find yourself doing uh group by operations or at least you should be um in a bunch of those data analysis tasks in the first part of that
[02:05:22] analysis tasks in the first part of that project um so I just wanted to review the group by operation just to see some examples of it. Um so um in this so I'm examples of it. Um so um in this so I'm in the 10.01 notebook but I am um I'm
[02:05:37] scroll down a little bit to this uh section 8 here uh of the notebook and um it goes through a an example with some fake data here. So we have this fake
[02:05:49] fake data here. So we have this fake data with um different a column for the data with um different a column for the category of product and then the revenue um from from those different products. Um
[02:06:01] and uh I just wanted to show off some of the group by examples. So when we load this data, we initially get this data frame and one of the operations we can do is this simple group by operation which allows us to group according to a
[02:06:17] column or even multiple columns if we want to. Um and so this is a good example of grouping by this um categorical feature. So this groups by categorical feature. So this groups by the uh this category column which is
[02:06:32] the uh this category column which is going to look at the um the category and look at all the rows for each of these unique values in that column. Right? So it's going to look at all the rows where we had the electronics, all the rows
[02:06:47] where we have clothing and then do something with that. So you're going to have some aggregation that usually follows the group by. In this situation, we only care about the revenue column. So obviously that's the only one there
[02:07:02] So obviously that's the only one there is, but um you know there could be other columns there that we aggregate. In this case, we're only aggregating the case, we're only aggregating the revenue. Um but uh in the in the
[02:07:15] project, this you know this may be the sales column, right? that that sales column that we generate from multiplying the price times the quantity. Um, and we generate a sales amount. Um, and so for example, like we could take the sales
[02:07:30] example, like we could take the sales and total it per category. That's what read this group by, right? Is you take the sum of the revenue column per
[02:07:42] category. So, what that's going to look like is um it's going to take each entry that's underneath the electronics category and add those together because that's what our aggregation is. But obviously, that math could change
[02:07:58] depending on what aggregation you put there. So, you have a lot of options. You can do things like mean, which would average all those entries per category. You could do median, take the median across all those. You could take the min
[02:08:12] across all those. You could take the min or the max. Um you could take the count which would give you how many records there are per or how many rows there are there are per or how many rows there are per category. Um which maybe which is
[02:08:25] essentially like a value count basically is exactly what that would be doing. Um so how many rows do we have where we have electronics category? How many rows do we have that have clothing? That would be if we had um count here would
[02:08:40] would be the equivalent of that. And there are use cases for count. Um actually in the first project with the Tesla data, I would recommend using this Tesla data, I would recommend using this kind of group eye to count how many um
[02:08:55] how many rows there were with the uh driver as the victim or how many rows there were with the uh cyclist uh or pedestrian that was in the accident. Um
[02:09:08] so these kind of group eyes can actually be really useful. Um but in this situation like the total sales uh um per this column is a really useful
[02:09:23] uh um really useful aggregation. By the way, this part of it here is an way, this part of it here is an important part in the sense that um what it does like if we didn't have this and I'll show you the effect of that.
[02:09:37] Actually, let me just show you the effect of not having this. So if we don't have this um every time you do a group by it puts your whatever you're grouping by as the index by default. So essentially
[02:09:53] what we're going to get is a new data frame that has this as the index like frame that has this as the index like each unique category as the index and then the column is going to be this aggregation of the revenue. So let's see
[02:10:07] aggregation of the revenue. So let's see that if I run this code, we're going to look at the original data frame and then the grouped data frame. We can see the difference of those two. And without that reset index
[02:10:22] And without that reset index um rerun that uh without the uh reset index um you can see we get
[02:10:34] this data frame. Now look at the index of this last data frame. It's the index is the category which is either clothing or electronics. But but that that may be or electronics. But but that that may be okay for some of our use cases. Um but
[02:10:48] this is now the index. It's not its own column. And it may be desirable to have that as its own column. So we can do things like filtering, right? things like filtering, right? Um but you know one of the one of the
[02:11:01] things that uh we can see is it definitely aggregated properly right so this got totaled um for the clothing was 300 + 400 which is 700 that makes sense
[02:11:13] that's the sum um but obviously we can see how that might change if I put in uh see how that might change if I put in uh max if I looked at the the max there um category that was 400 00 because that's the max and then across the electronics
[02:11:30] the the max is 700. So obviously the aggregation there matters but for the sales um you know the sum is likely what we're going to want is the total uh uh per day. So when we group by um
[02:11:49] the what are we grouping by is generally going to be the day but it not always the day. It could even be the store. So, one of the one of the suggestions or one of the things we want to do in the task is group by the store to figure out what
[02:12:04] is group by the store to figure out what is the you know the the revenue uh per per store. Um the other thing we could do is group by multiple categories. So
[02:12:16] in that uh in that project it may be in that third project it may be desirable to group by multiple things. If you want to group by multiple things, you just to group by multiple things, you just have to put it in a list. So, um,
[02:12:29] like this where we want to group by store and then group by day and take the total across the sales. So we should have a value per store per day and have
[02:12:42] have a value per store per day and have a total um which may be desirable to then we can aggregate that even further like we can resample it which I'll I'll like we can resample it which I'll I'll show us how to do in a moment um
[02:12:54] but but that may be something we want to do right per store per day. Um but what I want to show you too is the difference. So, if we run that reset
[02:13:07] index, um, what does that do? So, if we run the um, what does that do? So, if we run the reset index, um, what that's going to do reset index, um, what that's going to do is take this and move it out to a
[02:13:19] column. So, that's what reset index always does is it takes your index, puts always does is it takes your index, puts it over into a column, and then resets the index to be the default numerical index. So what's going to happen is
[02:13:32] we're going to end up with a new column called category and then we're going to end up with a default index of zero and one which are just the row index. Right? one which are just the row index. Right? So when we run reset index here that's
[02:13:45] going to produce our group by. But notice how we now have a default index notice how we now have a default index of 01, but we have a new column. about doing that? It's because we can do things like this now with the reset
[02:13:59] index, right? This will now work where I can actually uh filter according to this column, right? I can filter according to this column and and do I could even
[02:14:12] group by that uh further like in the case where um you know we're grouping by case where um you know we're grouping by date and store if I make that a column I can then further group by uh the date and resample according to that date and
[02:14:26] and resample according to that date and etc. Right? So it's useful to have often when we produce our groups it's useful to have that as its own column. So these to have that as its own column. So these are our unique groups here. Um,
[02:14:40] and so all you have to do is the reset index here will kick that in out into a index here will kick that in out into a uh into its own dedicated column.
[02:14:58] Okay. Uh we have some more. Um so Uh we have some more. Um so in the uh data frame so I think earlier in the uh data frame so I think earlier in this data frame we loaded a
[02:15:10] in this data frame we loaded a uh uh housing price data and we grouped uh uh housing price data and we grouped by the city and averaged the price per city. So that's exactly what this is doing is just grouping by city and um
[02:15:24] you know uh uh taking the price column and averaging it. So obviously what we could do is do different aggregations there like if we wanted to ask different questions like what is the max price per city um like in our in the project I
[02:15:39] think you are doing the max price per store for an item you would replace this you would group by the store and look at the cost and do the max right so it
[02:15:52] the cost and do the max right so it would be something like group by store ID and and then look at the uh um cost or what the name of the column is is the price and then take something like the
[02:16:07] price and then take something like the max which would um you know be the maximum uh price per store. Um so so that's something we could do, right?
[02:16:21] right? Um just as an example. So uh grouping by Um just as an example. So uh grouping by really really powerful and um you know it's something that is used for analysis all the time to generate uh different
[02:16:35] aggregations but uh across different columns. So uh in in that pro that third project you'll do a bunch of different group eyes. So this would be another one average price per city and and we can sort that too. So anytime you need to
[02:16:51] sort that too. So anytime you need to produce like a list of the top uh values, you can sort that according to remember the syntax oops remember the syntax for sort is to um for a data frame is to sort values and you pick a
[02:17:06] column. So this by argument is the column you want to sort by and then you can put um ascending to false to get a greatest to least sort. So the default is the least to greatest which is ascending equals true um by default. So
[02:17:22] you have to turn that off to do uh greatest to least there. So what that does so this was the city uh max. So every
[02:17:34] so this was the city uh max. So every city we got the uh max price.
[02:17:47] This was the uh average bathrooms per city. So this is all that housing price data, right? We just did some some group buy examples of it buy examples of it here. So really good examples
[02:18:04] and then we did some extra practice with this iris data set.
[02:18:20] may be valuable to go back and look at some of these plotting examples from some of these plotting examples from lesson five um within our data science course if you want some more uh examples of doing visualization
[02:18:33] because visualization could be really helpful to you know see the histograms helpful to you know see the histograms or see um you or see um you uh um different uh plots like a box plot
[02:18:45] uh to to kind of visually tell us the story that we're looking at in terms of story that we're looking at in terms of trends or uh or patterns or those kind of things. Okay. All right. So, this is a good
[02:18:59] resource for that third project. This this uh lesson 10 demo from uh the data science course. Any questions about the group by operation?
[02:19:12] I know it's been a while since we've done it, but there's some good examples. doing. It's a, you know, if you're familiar with querying data like in a SQL structure, it's literally the same thing. It's it's the same as doing like
[02:19:26] a group by and a query. Um, you think of this data frame as like a table. Um, that's that's all it's doing is just processing this aggregation. um that we would be doing in like a select statement where we're grouping by
[02:19:41] select statement where we're grouping by this this uh um this this uh um column in the table.
[02:19:55] Um I wanted to show one more that I think would be um helpful for think would be um helpful for the uh um third project which is going to be um resampling. So this is over in the 4.03 notebook in
[02:20:13] the data science course as well. So another good resource for that third project besides that lesson 10 notebook is this 4.03 3 notebook. Um because it has a section on resampling time series data which is exactly the kind of thing
[02:20:28] that you want to do for that third project, right? Is you have daily sales project, right? Is you have daily sales values and you uh want the ability to roll that up into quarterly, yearly, monthly, weekly, those different
[02:20:43] frequencies. Um and it turns out it's really really easy to do. Um the only catch to using reample is the
[02:20:55] the only catch to using reample is the date has to be um you know the index and date has to be um you know the index and so um it may not be the index in which case if it's not you can easily make it the index by um running an operation
[02:21:11] like this on the data frame that you have which will set the index to this. Now the the uh alternative to running set index is if you group by date
[02:21:25] um and return a result from that it will automatically make the date if you don't do reset index it'll make the date the index right if you if you imagine you um
[02:21:38] index right if you if you imagine you um so if you had a data frame that was uh so if you had a data frame that was uh and you did df.group group by and you and you did df.group group by and you group by the date. Um and then you did
[02:21:50] uh then you looked at the sales column and you did the sum. Uh this will this here would return a dataf this will this here would return a dataf frame result which is grouped by day
[02:22:06] frame result which is grouped by day grouped by the date. Um and so but this would make it so that the date is the index, right? So this would make it um this result df um this data frame would have the index
[02:22:23] um this data frame would have the index would be the date but because remember if you're not resetting it whatever you group by becomes your index. And so that's that's one way to get it that way. The other is to manually force the
[02:22:37] index to be the date by running this operation. Um, in case you happen to reset it for whatever reason and you have a date column, you can always make it that. Now, why is that relevant? It's because the date needs to be the index
[02:22:50] one way or the other so that we can run this reample function. So uh on a data frame on any number of columns that you frame on any number of columns that you want to you can run this reample
[02:23:05] want to you can run this reample which is essentially a group by at different time frequencies. It's essentially a different grouping by time frequency operation. And remember you can swap this over to
[02:23:19] And remember you can swap this over to be um whatever frequency you want. So you can put in month, you can put in week, you can put in year, you can put in quarter um and that will uh basically roll your
[02:23:36] data up into those frequencies and do some type of aggregation. So um maybe you want to do your average sales per quarter, you would use mean, right? Or
[02:23:48] maybe you want to look at your max sales per quarter, you would use max. or you want to look at your total per quarter, you would look at the sum. Um, different aggregations there. So maybe mean um and maybe you want to do this at
[02:24:05] mean um and maybe you want to do this at the year level or the weekly level, whatever that is. But um it's a really simple operation is just to run this resample and then you can do some aggregation off of that like an average,
[02:24:19] aggregation off of that like an average, a max, a sum, a count um whatever that a max, a sum, a count um whatever that is. Um but so really straightforward. It's um pretty easy to use is this resample function. And you you want to
[02:24:31] use that anytime you want to take your dates and roll them up into some aggregation. Like you want to take daily data that you have, which is what you would have from the sales if you group by date um and you want to get like a
[02:24:49] weekly average or a weekly total. And so what that's going to do is give you a new data frame that has index by the week and then give you the aggregation of that result like the the total or the average or the max or min or whatever it
[02:25:04] average or the max or min or whatever it is. associated with unique combination of values. Yes, [snorts] that's exactly what it does. Great ex great uh description. Yes, you're exactly right.
[02:25:21] It's exactly what group eye does. And to some extent, resample does the And to some extent, resample does the same thing. It's just oriented towards timebased fields like a date, right? It's really oriented towards that and
[02:25:35] and aggregating those to different frequencies. frequencies. Um, that it does. So the example we had, if you guys
[02:25:49] remember, like this was an example where we we had um fake data that was at an hourly level for a single day. So we had all this hourly level data with whatever all this hourly level data with whatever fake values are here. Um and what we can
[02:26:04] fake values are here. Um and what we can do is resample that to a day. So because this data is at a lower frequency, right? It's per hour, but maybe I don't care about all the data at the per hour level. I just want to know like what is
[02:26:17] level. I just want to know like what is the average of this across the day or what is the total of this across the day in which case I can resample to a day. What that's going to do is collapse all of these
[02:26:30] of these down to a single value for the day and and then it's just a matter of what am I producing? Am I getting a total? Am I getting an average? How am I reducing that frequency? So in this situation um
[02:26:47] we were reducing to the to the sum. So we resample and we uh um take the sum I believe which um totaled this up across the day and returned a single value for
[02:27:00] that day. So look at what the new uh uh index is. It's now a day. I don't care about the data at the hourly level. I've just rolled that up into a single day
[02:27:12] and taken the total. But obviously we could take the mean or the max or the min or whatever and get a single value for this and this um for this day. And the same thing works like if we had all this data at a daily level, we could get
[02:27:27] an average for the month or a total for the month, right? And that's what you want to do in that third project is really um you know that third project is really um you know generate um different quantities uh per
[02:27:43] uh uh per different time frequencies like quarter, year, uh month etc.
[02:27:56] to bring this one up is it's another good example for that third project of good example for that third project of building a model um off of this data. Um so this was our our bike rental data set. Um we did a capstone session demo
[02:28:11] set. Um we did a capstone session demo with this um together and this one also had a similar kind of uh style to it as that third project of um building some type of model but at an hourly level right to predict the rented bike count
[02:28:27] right to predict the rented bike count as a regression problem. Um so this is a really good um resource as well for the third project because you're doing something similar. you're trying to make a prediction at a yearly level. Um here
[02:28:40] we're making a prediction at an hourly level across these different dates. Um, so this was a really good one that walked through all of that, like looking walked through all of that, like looking at uh converting our dates to datetime
[02:28:55] um objects which we did and then generating some features like day of the generating some features like day of the week um which uh comes from this week um which uh comes from this datetime uh um you know the the DT uh
[02:29:08] attribute um and and that's something the third project I think also asked you to do right is generate a feature like day of the week, day of the month, um
[02:29:20] day of the week, day of the month, um uh um what month of the year is it? Um uh um what month of the year is it? Um so those kind of features. Here we generated a weekend feature which was looking at day of the week if
[02:29:32] it's greater than or equal to five which means it would be um Saturday Sunday which would be five or six um for the index. Uh and so we had a weekend feature and we looked at some of these in the correlation.
[02:29:48] We looked at some of the counts. Most of them were piled up here in the like 250 like 0 to 250 range were most of them. But there were some days where it was pretty big out here to the right. So it's an interesting histogram of
[02:30:03] So it's an interesting histogram of that. [clears throat] Remember we uh have seen this before with Seabor the pair plot which um plots like every it it remember what it does is it plots uh a scatter plot of every
[02:30:21] feature against every other feature and then on the diagonals because you know if you're it doesn't make sense to plot a feature against itself it does a histogram right on the diagonal. This is a really This was a really cool pot of
[02:30:35] a really This was a really cool pot of um all of this data uh here. So that that's usually a cool um a cool uh visual.
[02:30:56] So then we started uh going towards our um our model and in this situation um we um our model and in this situation um we actually didn't need to really roll it actually didn't need to really roll it up uh to any other frequency because we
[02:31:08] just looked at the count um uh we just looked at the the count um and some of the other as as our target, right? the count because it already was
[02:31:20] at an hourly level and that's what we were predicting it at. But the extension to this would be like if I wanted to predict it at a daily level, we could aggregate this further, right? So like we have an hour per day. We could group
[02:31:38] this by the day and take the total um which would generate like a total amount of bikes per day and then we could resample that to give us like a total
[02:31:50] per month or per week, right? And and then we could make predictions like then we could make predictions like that. So even though this data is already set up in the way we want to make predictions is like per hour, it's
[02:32:03] just build the regression which is exactly what we did. Um we built the regression by just using all the features and um
[02:32:15] by just using all the features and um using the target um and what we did is we uh train test split it and then we actually built a pipeline um where we did the standard scaling to all the numerical features which is what
[02:32:29] a regression. definitely want to, you know, standard scale all those features. And then if you're um any categorical features, we one hot encode. Um and then we have our pipeline which does that with a linear
[02:32:44] regression. And obviously um we could swap out that linear regression for other kinds of models like a like a random forest regressor project. So this is a good um this is a cool yeah
[02:33:02] this is a cool demo or capstone that is really relevant to that third project. Like I said the thing that you want to do is aggregate you you will have to do is aggregate you you will have to transform the data to get it to be in
[02:33:17] the different time periods that you need for the prediction. So luckily for us, this data already was set up in the hourly, which is like this is what we wanted to predict was the hourly level of rentals. But as I said earlier,
[02:33:32] there's nothing stopping us that could group by this date and total this up per day, right? And then we could predict at a daily level and then we could even resample that day to predict at a weekly level or a monthly level or a yearly
[02:33:47] level. Hopefully that makes sense, right? It's not too hard to extend this to that kind of prediction. Um here it's just doing it at an hourby- hour basis. So um we are predicting that. But obviously we could like total this up
[02:34:03] across the day um if we wanted to. So we could group by if we wanted to. So we could group by the date um aggregate this to a sum and the date um aggregate this to a sum and then resample that date to a week,
[02:34:17] month, quarter, year. Um I think you would end up doing all of those in the going to be built off of though is the yearly values. So you would roll this up yearly values. So you would roll this up to a yearly total and then um use that
[02:34:32] to build a model off of or maybe a monthly one. or maybe a monthly one. see what it says.
[02:34:48] Make a forecast for the next year. I think it's at a uh potentially at a monthly level though because you want to use or maybe at a daily level you could do that. You could actually just use a daily level and predict the days for the
[02:35:00] rest of the year. It's kind of what this um it's kind of what this is doing. um it's kind of what this is doing. predicting uh the the counts.
[02:35:16] um and daily features and we would uh be predicting or I should say hourly features and we would be predicting every hour for all the all the hours that are in this test set. Right.
[02:35:37] this? So this is a really this this is a good resource for the third project good resource for the third project along with that the 4.03 resampling along with that the 4.03 resampling uh from data science and then the uh um
[02:35:51] uh from data science and then the uh um uh the lesson 10 on the group by from the feature engineering and in data science. I wanted to um pick up in this lesson kind of at this point of it which is on
[02:36:05] page um 23 um talking about various agent frameworks and the reason this is relevant is because I think it'll be beneficial to see under the hood of how these agents are actually built. So I
[02:36:21] think we may have some experience working with these things but um it's useful to see sort of the elements that go into building them and there's a few
[02:36:33] different frameworks in Python that help us build agents and particularly multi- aent systems. Uh so we want to talk about that here. What are those popular libraries that help us build them? Um there's a bunch in the industry. Um
[02:36:49] these are I'm going to mention a few that are really really popular and then we'll spend today doing some examples from a couple of them at least maybe all of them uh if we have time but that's kind of the the plan for today. So let
[02:37:02] me um go through this and kind of outline what some of those popular uh frameworks are um which would be here on this screen. So the three major ones that I want to touch on and do some examples from are
[02:37:16] touch on and do some examples from are uh langraph, autogen and crew AI. So so at a high level what are these? These are Python libraries um that help us build agent systems, right? So, so think about um you know we
[02:37:32] have Python libraries to do a lot of AI things like we've had tensorflow we've had pietorrch we've had um scikitlearn we've had pandas we've had numpy those all help us manipulate data build models build neural networks these three are
[02:37:48] additional python packages that help us build agents and multi- aent systems so agents interacting and collaborating with each other
[02:38:00] with each other um and so you can build out these workflows or multi- aent systems in in these kind of uh Python libraries, langraph, autogen, crew AI and they all have their differences. So they all have
[02:38:14] um differences in their syntax. They all have differences in what they're kind of suited towards on what kind of workflow or agent system they're kind of geared or agent system they're kind of geared towards. Um, but all of these are widely
[02:38:27] used uh in the industry to kind of build out um agent systems. And so I do want these and then look at some code examples of each one of them if we get if we have enough time today um so that we can get some exposure to how do we
[02:38:44] build agents um see some of those uh features of them. Um I want to start with talking a little bit about langraph uh and and then I'll talk about autogen and crewi. So um lang graph is uh from the makers
[02:39:01] So um lang graph is uh from the makers of lang chain which is another popular um LLM library that that helps you build like LLM based uh chat bots and those kind of applications. Um and it's an open source library. All of these are
[02:39:15] open source library. All of these are open source libraries in Python and um open source libraries in Python and um langraph is called uh lang and graph because it brings together LLMs in this kind of graph structure almost like a
[02:39:28] kind of graph structure almost like a tensorflow in in some ways to um build out the multi-ent system or the or the workflow. Um and so langraph utilizes nodes and edges to connect like a node you can think of as an agent or a
[02:39:43] function. So doing some type of task and then you can connect that to some other node which could be another function or an agent. You can connect those with edges. You can have conditional edges. Um so essentially you can build out your
[02:39:56] workflow especially if it's very linear and sequential. that lang graph works really really nicely to connect um like I have one function here that passes it it does something maybe it does like an internet search pulls in some some
[02:40:10] research and then passes that to an agent who um summarizes it or writes a report or something and then you can send that on to maybe a document reviewer reviews the results of that and then u
[02:40:24] maybe make some edits to it um and then you can kind of uh pass along the final kind of the final result of the um review. So, lang graph allows you to review. So, lang graph allows you to build an um an agentic system that's
[02:40:39] build an um an agentic system that's very graph oriented and and um flows from nodes uh connected by these edges. And so, if we have some time, we'll see how we can build that. But um Lang graph is very suited towards agentic workflows
[02:40:53] that are very sequential in nature or or laid out kind of in sequence going from not limited to that though like there could be situations where um we can have conditional edges. So maybe we go from
[02:41:08] point A to point B or C depending on a condition. So we do some research depending on the kind of data we pull back. Maybe we pass that on to um a content writer. Maybe we do like a another research if we don't pull back
[02:41:23] the right kind of information. Um and then those can kind of flow this way and then those can kind of flow this way and then they can be um we can bring back together the results in kind of a graph like this. So it does it need to be
[02:41:35] completely sequential? No, you can Langraph has a lot of flexibility to build many different kinds of graphs. But um that's the way that library kind of operates is all graph oriented. So you you build out your graph and then
[02:41:48] when you run the workflow you kind of launch the graph and then the um you usually send in a prompt to this first node um to say here's what you want to node um to say here's what you want to do and then um then this kind of kicks
[02:42:01] off does something passes the result to the next node that does something passes the result and each node can be like a specialized agent or a function. So, lingraphph is kind of designed to work like that. Um, and if again if we have
[02:42:15] we'll do some demos with it, build out some examples with it, and I think it'll be pretty cool to see that one in action. Um, so that's one way of approaching building an agent system is kind of modeling everything on a graph
[02:42:29] structure. These other two, so autogeneri don't really do that. they actually um they they model they both model an
[02:42:41] multi- aent system in in sort of a a team or collaborative fashion. So what team or collaborative fashion. So what autogen does is it allows you to um build multiple agent objects and then kind of put them together in a team. So
[02:42:55] you um have an agent and then you have maybe another agent and then they just maybe another agent and then they just um they conversate with each other. So kind of put them together inside of what's called a group chat structure and
[02:43:09] they they pass ideas and collaborate amongst each other. So, you may have like a um a planning agent and then you can have maybe a writer agent who's writing up an itinerary or something and they'll work together by having a chat
[02:43:24] uh amongst themselves and that kind of plans out the work that each one will do plans out the work that each one will do and then they'll um uh agree like the planner will go out and find details of the trip and then the the writer will
[02:43:37] kind of write out the itinerary. Um but but autogen works by instantiating agents and then putting them together in kind of a chat structure. That's how time, we'll do examples of all three of these so you can see the differences.
[02:43:53] these so you can see the differences. But um that's very similar to crew AI, which is uh the reason it's called Crew AI is is um because you are doing agents and putting putting them together in a crew. So you build an agent and
[02:44:09] then an agent and then maybe another agent um which are all objects that come from the library. So instantiating agents is really easy. Um and then you basically put them together in a crew. So you have a crew of agents and then
[02:44:25] the crew kind of um similar to autogen kind of conversates together, works together to uh figure out to the planning what to do um which tools to use those kind of things and then um goes out and kind of
[02:44:40] uh executes the task according to that planning um depending on what your initial prompt is. So these two are very very similar in in the sense of building out agents and then having them work together kind of uh autonomously.
[02:44:56] Langraph is a little different in which you are generally supplying the structure of the workflow. And so again the langraph tends to be more oriented the langraph tends to be more oriented to building an agentic system that is uh
[02:45:09] is. You you know it should go from A to B to C. Um whereas ogen and kay are kind of just instantiating agents and then having them go out and do things uh on
[02:45:21] having them go out and do things uh on their own as part of a team or or a crew in the case of crew AI. Um so again we'll do examples. I think the very first place that we'll start uh after we do a little more discussion of these
[02:45:34] three frameworks is my plan is to do some crew AI examples and then we can work our way back to Langraph um and then see if we have some time for autogen. If we do cry it a lot of it's going to be similar to if we had done
[02:45:48] autogen they're very similar um but again these are all open- source uh Python libraries for building agent systems. So they're, you know, very very popular. Um, and, uh, you know, a lot of
[02:46:05] popular. Um, and, uh, you know, a lot of people, um, will build agent systems in in any one of these three. And then, um, there is the possibility to have an actually talk to an agent system that's built in CURI. They can talk through
[02:46:21] what's known as an agentto agent protocol. Um, that's possible. um we won't really see that today, but uh if you're interested in in exploring that, that is um kind of like MCP we talked about last time, but um that's known as
[02:46:34] A2A um protocol, which is uh very communicate with each other even if they're written in different frameworks, they're written in different frameworks, which is interesting.
[02:46:48] Okay, so again, just a high level summary of those. We could spend a lot of time um and if we were in a full, you know, the AGS program, we would spend time on lessons going through each one of these. Not going to get all that time
[02:47:01] of these. Not going to get all that time today, but I I hope to establish a couple examples in in each of these if we can. I think it'll be valuable to see um how these systems are kind of built and especially what are the differences
[02:47:15] between these um Python libraries that can help us build these agent systems. can help us build these agent systems. That'll be kind of the goal of today. That'll be kind of the goal of today. Okay. So,
[02:47:31] uh let's keep going with some of the differences. So, uh in terms of some of differences. So, uh in terms of some of the uh um scaling and optimization, now in in terms of a lot of this is infrastructure setup. Um so you know
[02:47:46] depending on if you're doing that part of it may not be that interesting. Uh could be another whole another team managing kind of the uh infrastructure or devops of this. Um but they they are these libraries are
[02:47:59] different in the sense of how they are scaled or how they are optimized. scaled or how they are optimized. Um so there can be different ways to Um so there can be different ways to manage their scaling. Um
[02:48:12] uh what I really wanted to focus on is how they manage state which is how they manage state which is important. Um essentially so the state management is really important because that uh enables sharing information from
[02:48:25] agent to agent or throughout the workflow. um you know all the you may have a team of several agents um or in in Langraph's need to share information between those nodes right in order to pass um context
[02:48:42] from agent to agent to agent um and so they handle each one of these frameworks they handle each one of these frameworks handles that slightly differently um handles that slightly differently um so uh in langraph there is a notion of a
[02:48:55] of kind of a managed or version state Um and it's kind of like a snapshot of all the context that's currently being managed. Um so again if we do a langraph managed. Um so again if we do a langraph graph example we will see explicitly um
[02:49:10] that state object kind of being manipulated and passed around which is the way of sharing information from node to node to node. So if we have a node, um we may have a desire to have the
[02:49:25] result of the first agent or the first node, you know, that context pass on to node, you know, that context pass on to the next one in the workflow. Um and so the next one in the workflow. Um and so that's that's done through this um state
[02:49:37] langraph. Um, but each one of these frameworks does that a little differently. Like Autogen, as I said, has a team that's conversating. So they tend to have in the session,
[02:49:51] that's maintained throughout the session that each agent has access to. So it's kind of like a um almost like a database or like an internal memory um that every
[02:50:03] agent can kind of draw from in terms of the current conversation and any context conversation. Um so so that's kind of how autogen does it. Um Koreai has something similar except it it um instead of having that
[02:50:19] in memory it can save it off to uh a cache um something like uh reddus which is uh just a essentially a data key value data store um where it can save the state and again the whole purpose of that is to maintain both of these really
[02:50:35] that is to maintain both of these really is to maintain memory and context. Um is to maintain memory and context. Um so um in order to in order for each agent to have access to the conversation and kind of the chat history um as those
[02:50:49] are as those agents are working together in this team or having this chat uh in in autogeneri and again that's because system these agents are going to be working together they're going to be
[02:51:02] passing information whatever the result of one agent is would be used in the next one typically and they need to have access to that. Um, and so you need some way of kind of sharing that memory or that context. Uh, and they each one of
[02:51:15] these does that kind of in a in a slightly different way. So just pointing optimized in slightly different ways. They're just different Python packages. So they're managed slightly differently on that front. I don't want to spend too
[02:51:29] much time going into those because again they're more DevOps topics. Um but relevant for us I think is the state management which is important for agents that sharing of information right from agent to agent to agent I think that's
[02:51:44] relevant it's and just to say it's handled slightly differently from from handled slightly differently from from uh library to library
[02:51:56] across as we go so all these things we may come across as we go through you know [clears throat] examples today of them and get some exposure to them. Again, the goal for today not to come away as experts in any one of these.
[02:52:09] It's more to just get some exposure to to each one of these and see, you know, how agent systems are truly built underneath the hood.
[02:52:24] compliance and observability um each one of these has uh um different ways it can handle authentication and things like uh keys and rolebased
[02:52:36] things like uh keys and rolebased authentication. Um so each one can can uh do that differently. Again, I don't want to get too far in the weeds on that part of it because it's more of this the DevOps setup of each one. Um, what is
[02:52:51] interesting is the observability. So, what I have noticed myself from each one of these these libraries is they have very good logging. Each one of them
[02:53:03] have very good logging. Each one of them has good logging. So, you can see has good logging. So, you can see um in detail the uh various operations of each agent. So you can see every time it calls the LLM,
[02:53:15] has a conversation with another agent, um every time it uses a tool, what the result of that tool uses us usages and you can integrate that with other and you can integrate that with other libraries like open telemetry or um
[02:53:30] libraries like open telemetry or um different like uh log exporters um or different logging tools like a data dog or elastic search or things like that um to get really good observability into what your agent system is doing, but at
[02:53:43] a minimum they all have really good builtin logging. Um, which is important. And so for agent systems, that's really really critical so you can see what debugging them, right? So you can see what kind of LLM calls are they making.
[02:53:57] Are they making redundant calls way too often and and driving up cost because that that of course costs money um whenever you're using tokens that way. and also what kind of tool usage is it
[02:54:10] tools, is it editing the right files, is it accessing the right data that it needs to all of that you can kind of um see from the logging capability. Uh, so what I've seen is each one of
[02:54:23] these has really really good built-in logging and they hook up to other um kind of logging tooling that's out there like a data dog or um elastic search like a data dog or um elastic search those kind of things and open telemetry
[02:54:38] uh for kind of tracing instrumentation. So again, like setting all that up, not because that's more DevOps stuff, but just to say they have really good built-in logging to see um a lot of the details of of the what's going on with
[02:54:54] the LLM and the tool usage from the agent perspective.
[02:55:11] those uh frameworks. Again, one of our um goals is to just go through each one of those and in and some demos and see some of those examples. Um but so we may be wondering why do those exist? And part of it is for efficiency.
[02:55:28] So if we have um a lot of our agent systems built out in those frameworks, we have the ability to reuse them um for automation and that helps a lot of business needs, right? So we're
[02:55:43] building our agent systems for specific business impact. Um we don't need to those components already built out in one of those frameworks, we can kind of one of those frameworks, we can kind of reuse it. Um and that that it also gives
[02:55:56] reuse it. Um and that that it also gives us a good um starting point for managing um our agent systems. We don't need to start from scratch and do bare bones Python code. Um when we use the frameworks, we get a lot of benefits. We
[02:56:09] get like builtin agent tooling. We get builtin um logging and observability. builtin um logging and observability. Um, so there's a lot of uh a lot of that we don't have to do from scratch, which is nice and allows us to reuse
[02:56:24] which is nice and allows us to reuse these things. Um but uh it you know the question always comes up which one of these frameworks should I use and it what I would say to that is it mostly comes down to um preference and also I'd
[02:56:40] comes down to um preference and also I'd say two things preference and alignment of tasks. So depending on what you're trying to automate, it may make more sense to use something like a lang graph. If it's very very
[02:56:52] sequential and ordered and you need things very sequenced um and it's very structured, I'd say langraph is a great choice. But if you want more open-ended choice. But if you want more open-ended conversations and deliberation and maybe
[02:57:06] um task delegation, something like a career or autogen might be better need to necessarily be in a predefined sequence and it can kind of go back and sequence and it can kind of go back and forth um than those might be better. So
[02:57:19] which one to use sometimes does depend on the workflow itself. Um other times you can get away with do using any one of the three. um just comes down to preference almost like which one do you use TensorFlow or PyTorch remember that
[02:57:32] kind of comes down to preference might also come down to what have you done in-house before um if you go to to a company that has you're probably just going to stick with crew AI right if they've already done a
[02:57:47] lot in line graph probably just going to stick in lang graph unless there's a good need to uh switch those so um a lot of it comes already have because you can reuse them, right? You could you could stay in that
[02:58:00] right? You could you could stay in that framework and kind of reuse them. right one in terms of lang graph in particular um it's helpful for uh
[02:58:15] workflows that are very sequential as I said they may have dependencies from task to task that we need to have access to like the result of task A pass that on to task B do that one pass that on task C etc. Um, it also does support
[02:58:31] repetition. So if you need to do looping, um, you can have a graph that looping, um, you can have a graph that has a a looping edge. So you can kind of has a a looping edge. So you can kind of redo operations as you need. Um,
[02:58:43] and so here's some examples on on this slide of some use cases, things like loan application processing where you are you have a very structured workflow, verification to an eligibility check to
[02:58:56] to a loop if things are missing and then maybe an escalation uh if if fraud is detected. All of that could be a graph and we could build out that graph. Um in a very structured way uh so that's that's a good use case for that. Um
[02:59:12] patient intake again that's one that goes from kind of point A to point B to point C. We collect symptoms, suggest tests, handle missing data, um pass off to a human in the loop, you know, doctor. Um
[02:59:26] polite disruption handling. So maybe like an automation workflow there for handling um delayed flights, rebooking, rechecking preferences, um sending notifications. Again, very structured workflow. Um and
[02:59:40] it's a good candidate to be put in this graph um sort of structure in in a ling graph. Okay, so just some examples. Of course,
[02:59:52] there's many more, but those are just some thoughts and uh again we'll hopefully do some demos today where we see uh some interesting examples, too.
[03:00:06] talked about with collaboration and building a team where they can kind of debate um conversate, have multi- aent conversations. And this is really good for things like ideas, um brainstorming, um uh uh code
[03:00:23] review, um these kind of things that you want to build a consensus around or have some debate over. Um so maybe like a bug debate over. Um so maybe like a bug fixing um code review agent uh workflow.
[03:00:37] So we need to propose a fix. Uh we have a tester agent that can run test cases. Um an architect that can review design. Uh and then they can kind of conversate solution. So um something like a code review agent
[03:00:53] So um something like a code review agent would be uh you know a good use case for planning. That's another one. Again, you're going to debate. You're going to have multiple agents set up in your team. They're going to conversate. Um
[03:01:06] they'll do that all in kind of this group chat setting. Um complaint resolution for customer service. That's another one. So, review service. That's another one. So, review the complaint. Um verify all the rules.
[03:01:19] Um legal can check compliance. and you have these different um agents that can work together to try to figure out how to resolve the customer complaint.
[03:01:34] Uh how about sched scheduling staff across multiple stores? we identify the case in which uh will benefit from AI versus oldfashioned
[03:01:49] automation framework? Um, so that's a good question. I would say uh it where you're going to benefit from AI is um where wherever you want to use the
[03:02:04] capabilities of the LLM to plan and think autonomously. So in in other words, like you don't want to define all of the rules, you don't want to define all of the solution framework. you just want the agent to be able to think on
[03:02:20] its own and come up with the solution. Um so in a situation like that um where again you're not prescribing all the details you want the LLM to think on its own maybe given some context so like
[03:02:34] some guardrail context ultimately think on its own to come up with a solution um on its own to come up with a solution um that may be a good case uh that may be a good case of where you don't need to uh you may be better off using an LM in
[03:02:49] something where like you have very specific rules tools and you don't need specific rules tools and you don't need an NOL to think and plan out a solution. Um, and you you can just get away with using regular functions and it's it's
[03:03:02] very deterministic. Um, and you don't need any planning done. You don't need any thinking done. You don't need any um kind of planning and tool usage combined together. Um, that's where uh oldfashioned automation
[03:03:17] probably comes into play. So again, the thing that AI brings to the table, thing that AI brings to the table, generative AI, is the LLM like planning process, acting very autonomously. You're not really in the loop very much,
[03:03:30] right? You just provide the prompt and and which is the task and let it go. you're really gaining from the generative AI is the planning um aspect
[03:03:44] and the thinking of what to do on its own rather than you providing the rules and the direct kind of do this then this then this then this and this which is then this then this and this which is all codified and kind of rules.
[03:04:04] probably the best example I have is with like a coding agent like a like a cloud code or a codeex or something where it can plan out what to write. So it can it can plan out like a design and then do an implementation of that design.
[03:04:21] So you can give it a very broad task like build me an app to uh find the nearest restaurants within five miles of me and uh aggregate them onto a screen with all the reviews and whatever you know whatever features you want. It can
[03:04:36] order to write the code, it's going to have to plan all that out, right? So you're that's where you're really utilizing the LLM to um like automate the coding process. It's thinking it but it has to plan that out.
[03:04:51] it has to plan that out. So it has to first think of what you know what steps do I have to take? Um first I need to build like these various functions and then I have to fill them in and then I have to write tests and
[03:05:04] then I have to integrate them then I have to build the UI right it's going to plan out all those steps and the LM is going to do that autonomously uh or the agent is um so that's that's
[03:05:20] thinking what steps do I need to do to complete this task and all you're doing is providing the general task like build me this app and it it will think about all the steps it needs to take to complete that.
[03:05:34] right? You're not in the loop on those steps. The LM is planning it. Is a model trained to build that process? Yeah, the LLM is so um
[03:05:47] absolutely the LLM is trained to do that. That's what the model uh like the the large language model has been trained to provide responses to those prompts which are you know build me an app and part of that is um planning out
[03:06:02] what that what that looks like. Um and that's yeah the model has been trained that's yeah the model has been trained to do that trained against a lot of data. So it's one of those like transformer-based
[03:06:17] one of those like transformer-based large language models like a claude um large language models like a claude um you know sonnet or opus very large large language model that's been trained against a lot of examples a lot of code
[03:06:29] bases like think of like all of GitHub um in order to be able to design code so usually you can you know use something off the shelf you probably
[03:06:42] wouldn't need to um retrain [clears throat] a model from scratch to do that. So I wanted to um talk about crew AI a autogen and langraph. Some examples for crewi might be doing like market
[03:06:57] intelligence. So we uh again have multiple agents in the mix in our crew. So kind of like autogen we have a team. They chat amongst themselves. Crewi we have a crew which is a team um and there's multiple agents in the crew. So,
[03:07:11] we have a researcher agent that's responsible for gathering um competitor data, an analyst agent that's looking at different trends, and a writer agent formatting the final report. Um, and so they work together in the crew, but
[03:07:25] inside of that crew, there's kind of chat going on to delegate um to plan and delegate like who should accomplish what task. Um, so there's an overall prompt that from a user saying, I want to do market intelligence or
[03:07:39] build a market intelligent report around this product, let's say. And then the crew has to figure out how to get that done. It's going to conversate. It's going to um uh uh plan out how to do that and then
[03:07:54] basically delegate and figure out um based on the the various agents who's best at doing what task and then going out and accomplishing those tasks. So place we're going to start with some of our demos. I want to start on Crew AI.
[03:08:09] We'll see some demos of this where we build out multiple agents. Um the going to be the second one, this travel itinerary planning. So we'll have a destination expert that's responsible for identifying attractions to visit. Um
[03:08:25] we'll have a budget analyst to optimize costs and a scheduler to um basically costs and a scheduler to um basically build out the itinerary um iteration. Maybe it actually has the right tools to um do the bookings
[03:08:40] itself. So maybe it integrates with you know various APIs like on like on a travel website and it could actually you know do the booking itself. But the the demo that we'll do um will just be writing up the itinerary.
[03:08:54] um not actually you know doing the booking but that could be a real use case is actually uh arranging the booking. incident response team. So again this is a team or a crew. We have multiple
[03:09:09] agents here. A threat detector to locate anomalies investigator to look at logs and communicator prepares a summary. Um, and these all these agents all work together in the crew delegate, figure out who's best at what based on some of
[03:09:24] their background information about those agents that we're able to set up. You agent's really good at anomaly detection. This agent over here is really good at um looking into logs. And this agent over here is really good
[03:09:37] at just writing just writing in general or writing summaries. And we would delegate out, you know, the task to the appropriate agents. plan it, plan and then delegate. So again, we'll see that we'll see how to set up these agents.
[03:09:50] We'll see how to put them together in a team um and then basically execute them team um and then basically execute them on a task. So we'll see that in some of our code, but just going back real quickly, you
[03:10:05] know, in you can see the differences. The the lingraph is going to be better suited to things that have a lot of structure already. So we want things to structure already. So we want things to go in a very specific order um and and
[03:10:19] it's easily modeled as a graph. Those are the best ones for langraph. And then kind of autogen and career are better for team oriented collaboration um that doesn't need to go in any one particular order. But we do have the ability to
[03:10:33] kind of delegate um and debate when needed um between those two. So uh you can kind of see the differences there.
[03:10:49] So, uh that being said, wanted to talk about the some of the challenges and uh where LLM agents, LLM based agents are kind of going. Um and then as I said, we'll take some time to start diving into demos with them. Um but wanted to
[03:11:07] wrap up this lesson which is more of an introductory lesson to agents by talking about some of the challenges. So uh the first place to start with um these agents is they are very powerful right they can help automate things and
[03:11:23] help make our life easier. Um but they do have a lot of implementation hurdles do have a lot of implementation hurdles and challenges that are um very real. Um
[03:11:36] and challenges that are um very real. Um one of them is the um basically the performance bottlenecks that we see with these agents and a lot of that is tied these agents and a lot of that is tied into the LLMs themselves, right? because
[03:11:48] into the LLMs themselves, right? because LLMs um take some time to generate uh responses and to do planning and especially in their working in a team to conversate and delegate that all takes time.
[03:12:01] time. So um and it's not deterministic, right? So um and it's not deterministic, right? So there can be a real um performance bottleneck issue that that even if you're using the latest and greatest and
[03:12:15] you're using the latest and greatest and kind of fastest models um it's it's there's still going to be um latency concerns generally whenever you're using these agents because um they are going to take some time to come up with the
[03:12:28] solution, plan it out and then do and then act um and then maybe iterate on that and there there can be uh latency concerns especially with really really something people are always thinking about with these agents is like the time
[03:12:44] you're doing something that's very very timesensitive like trying to identify fraud and flag it and escalate it. Um you want to be really wary of of time or
[03:12:56] you're doing something like stock trading or something. Um so these agents are very um you know sensitive to latency in some ways but um so that that's always a concern uh even down to like a you know
[03:13:11] day-to-day concern of just interacting with LMS we are probably aware that you know generating a response can take time depending on the model. Um there's also uh similar to that there's scalability issues. So um it can be a challenge to
[03:13:29] manage these uh workflows especially from a DevOps perspective and infrastructure perspective. Um especially if you want to use your own LLM and not have to rely on paying a provider for the LLM. Um you know that
[03:13:44] can be a real concern is how to scale out our workflows, how to scale out our LLM so we can use them in many different situations with our our agents. Um,
[03:13:56] and so a lot of organizations are facing those kind of challenges of getting up those kind of challenges of getting up and running with like a a self-hosted model for instance or fine-tuning that and using a self-hosted model that's
[03:14:08] fine-tuned. Um, and then making sure it's the infrastructure is there to to different agents that are utilizing those LLMs. so that can be a challenge is just the fact of just managing the LM
[03:14:23] fact of just managing the LM infrastructure to use it with agents um uh that maybe interacting with LM very frequently and and you may have lots of people using agents, lots of workflows um that can lead to real scale
[03:14:37] challenges, right? Um the other uh concern is with any generative AI is uh bias in the training data. So that's that's not unique to
[03:14:51] agents. That's true with anything especially with LLMs that you know are especially with LLMs that you know are being trained uh to generate responses and there's a concern that they could provide biased responses or it biased or
[03:15:07] provide biased responses or it biased or even false answers um and make mistakes not perfect. It's not deterministic. It's not always the right answer. Um, so there's concerns there of of answers being biased or just flatout wrong,
[03:15:21] right? If they're coming from, uh, an LLM, uh, especially when working with agents, right, that are using the LM to kind of plan and and do things. Um, we don't want those to be biased. Uh, for
[03:15:35] agents in kind of these automated workflows like loan processing or scholarship processing or those kind of things, don't want them to be biased. and that's true across it's again not unique to agents true across a lot of
[03:15:51] facets of AI but certainly true here because LM are at the forefront of agent operation right they're a key key part of agents.
[03:16:08] see with the future? Well, we see agents evolving um every day. It seems like they're getting better and better. Um in certain areas like autonomy um for sure, like being able to do things um without human in the loop really at all. Um
[03:16:24] human in the loop really at all. Um uh being able to uh basically learn from uh being able to uh basically learn from their mistakes uh in real time uh with human feedback and then being able to take that forward and apply that to
[03:16:36] future tasks. That's a huge um thing we kind of see and expect over the over the kind of see and expect over the over the course of the future. Um course of the future. Um we also uh expect more domain specific
[03:16:50] uh learning. So as you know agents are being deployed in different domains. Um they're able to learn from those domains and they don't have to be explicitly fine-tuned. They can kind of learn from being exposed to data. It's what we
[03:17:04] better and better at operating in those domains. The more they're being exposed to tasks and questions and data from those domains, they just get better at handling that. Um, that's the hope and
[03:17:18] they don't need to be fine-tuned. Um, Um, and uh, again, so, so kind of that's what we're getting at here with the human AI symbiosis point at the bottom
[03:17:30] is the more we use it on our data, the more we expose it to our tasks, our workflows, our data, the more it can kind of uh, pick up on that without having to be explicitly uh, retrained and fine-tuned.
[03:17:44] um we can kind of integrate that into the architecture so that um it learns from that context o over the course of usage um and preser learns to preserve that context essentially that's we expect to see we also like as hardware
[03:18:01] gets better and the models get better we expect there to be more uh efficiency gain so the so the latency becomes less of a concern um and we've seen some getting faster at returning responses and um we've seen
[03:18:17] that over the years. Continue to expect that to to be true um o over time. And there's there's thought that um we're kind of hitting a plateau on the
[03:18:29] models themselves aren't going to get that much exponentially better, but the infrastructure of serving the the results and the actual processing of information gets faster. So you can you can uh return uh answers faster.
[03:18:51] Okay. So just to recap all of that lesson um you know we're talking about lesson um you know we're talking about these uh LLM based agents that um what makes them unique is their ability to operate without human intervention.
[03:19:04] especially two main things, right? Is using an LLM to think and plan and then having access to tools to take action on its own. Um, and we say plan and think,
[03:19:17] you know, that involves what tools is it going to use, what steps is it going to take to complete a task, those kind of things. That's all driven by the OM. cases for agents. Um and what we're going to see is
[03:19:32] there's you know some of those use cases can be modeled very effectively in these libraries like langraph autogen crew AI um to build out these agent systems to um to build out these agent systems to to accomplish those use cases. Um all
[03:19:46] we talked about bias security scalability um scalability um and uh so not you know part of that um can be tackled through uh logging and observability so kind of monitoring what
[03:20:03] the agent's doing how it's process how it's making LLM calls what kind of tools it's using um and then uh understanding that so maybe we can make it more efficient and optimize it make less redundant LM calls, those kind of
[03:20:17] redundant LM calls, those kind of things. Um, but in in the future, we expect the models to get faster. We expect them to improve and learn better. And um, we expect generally for them to be less biased and more correct, you
[03:20:32] know, as they're being uh, tuned and um, updated with new data all the time. Um, we expect them to get better at reasoning. So I want to talk about Crew AI and give an introduction to that
[03:20:45] AI and give an introduction to that library to help us um build library to help us um build examples of uh agent systems and and agentic workflows. Um,
[03:20:58] so what I'm hoping to get out of this is uh basically explore crewi especially some familiarity with how we build agents, how we put them together in the
[03:21:11] crew, how we give them tasks, how we give them tools that they can use, um, and then see how they interact together in the crew to accomplish those tasks.
[03:21:23] and then we'll do that over the course of some demos throughout. And the main uh application that we have in mind for this demo is a trip planner itinerary uh
[03:21:35] this demo is a trip planner itinerary uh agent. Um so we will uh again be doing agent. Um so we will uh again be doing that through multiple agents um uh in in this system to plan out help us plan out a a trip.
[03:21:48] So we'll see that over the course of some demos. Um so going back to just some background information about Crew AI as a package. Um again it's it is a library that helps us um build and manage a team of agents.
[03:22:05] us um build and manage a team of agents. So a multi- aent system. Um and each uh agent that we build as part of the crew. So the the team of agents is known as So the the team of agents is known as the crew, hence the name crew AI. Um
[03:22:19] each agent will have access to various tools that we supply to the agent and they'll also have uh metadata that guides them on what their specialty is and kind of what their goal is and what their tasks are. Um and that's helpful
[03:22:36] their tasks are. Um and that's helpful to know because um you know each agent that we uh define as being part of the team has the opportunity to be uh unique. It has the opportunity to be specialized. It can use its own
[03:22:51] specialty LLM for instance that has been fine-tuned. Um, and so each agent can be highly customized. Um, in terms of, uh, its specialty and what it's designed to do.
[03:23:06] Um, and so, uh, that'll turn out to be a big deal is providing that metadata for the agent when we initialize it um, as part of the crew of kind of what is it doing, what is it supposed to be focusing on um, and what is its role in
[03:23:21] the team. So kind of like no different than a team of humans. What is everyone's role? And so we'll define that um uh as part of the part of the crew. Okay. So that'll we'll see that in the demos. Um but that's the main idea
[03:23:38] is we crewi is oriented around the idea of building the crew which is just made up of a team of agents. Those agents have tool access. They have LLM access. They have uh roles and they have metadata about that that kind of orients
[03:23:54] them to what they're supposed to be focusing on. Um and again, it's a lot of it's going to come down to what is the syntax of putting that all together, right? What is the Python code look like that that puts all that together? And
[03:24:07] that that puts all that together? And that's kind of what we want to see. could imagine putting together a crew that is building an app. Um so we have
[03:24:21] maybe a researcher agent who's um doing some background on what the best uh packages are for that or what the what the design should be. We may have a code writer agent who's writing out the code and then we have an engineer who may be
[03:24:36] and then we have an engineer who may be testing it uh architecting it um those kind of things. But basically our our way we would approach building a system for this is um instantiating multiple agents. So these would be our agents
[03:24:50] here and then putting those agents together in a crew. So we have one agent, we have a second agent and then we have a third agent. We put this agent these agents together inside of a crew and then let the crew kind of operate on
[03:25:06] this prompt. So there there's a user prompt that's kind of launching the crew. Um and then the crew will delegate and and plan out and figure out what's the best way to approach accomplishing that given that we have access to these
[03:25:19] three agents which have you know various tools LLMs. Um, and as I said earlier, tools LLMs. Um, and as I said earlier, like each one of these agents could have um different tools. In fact, most of the time they will. Um, like the writer may
[03:25:36] have tools to edit code files. Um, the researcher may have like internet or Wikipedia search API tools. Um, the engineer may have um like execution tools to execute code and and debug it. Um, so each one can have different tools
[03:25:54] and each agent can also have different LLMs that they can use. So as I said, it gives us the ability to specialize each one like this one may specialize in in
[03:26:06] codew writing. This one specializes in research. It can use a completely different LLM that's been fine-tuned to do that task. Um, do that task. Um, and so the the other um thing about
[03:26:19] these frameworks and what's really nice about them is when we build an agent for this crew, there's no reason we can't copy and paste that code and use it in a have a different workflow we're trying to automate. Um, and that would require
[03:26:35] building a different crew, but we can kind of copy and paste this and use this in some other crew. um and we can use the engineer maybe in some other crew, right? And so um uh that's that's um one of the nice
[03:26:51] benefits of this framework is if we build out another Crew AI app uh or a Crew AI frame uh workflow, there's no reason we can't reuse some of the agents we've um initialized in other crews. We could mostly reuse that, which is nice
[03:27:06] could mostly reuse that, which is nice from a code perspective. it could. Um, and can it use other models?
[03:27:20] and can it use other models? Um, in theory it could. Uh, it could use Um, in theory it could. Uh, it could use a model as a tool. So, um, we maybe we have a need to make some type of prediction
[03:27:32] um, and run some sort of inference. um in which case maybe we are hosting our model on some endpoint and we we process some data and pass that through the some data and pass that through the model as as a tool. Um that's possible.
[03:27:46] Um and it is possible for an agent to use multiple LLMs. Um it's not very common I would say. Um, usually it will choose you either the
[03:27:58] user will choose one of them. Like the agent could have access to multiple but only use one of them when it when it's actually being used. Um, it's rare that it would need to utilize multiple LLMs. Rare, but um, you know, in theory
[03:28:14] Rare, but um, you know, in theory possible.
[03:28:27] which would be like having a like a prediction tool. And so you can imagine building out like a a fraud detection workflow in which case you have one agent like gathering trans like maybe parsing transaction
[03:28:40] data, gathering it and then passing that to or or using a tool which is sending that data along to the model, generating a prediction and then some way and and does something with that, right? maybe escalates it or
[03:28:57] um or sends an email or a Slack alert or something, right? So, so that totally could be possible where using a model is is equivalent to using a tool if that makes sense. Um very very possible.
[03:29:20] um I'm going to show you how to build a custom tool. So it could be that part of the custom tool is is running a model on some data. That's totally possible. Um like a a deep learning or a machine learning model or something like that.
[03:29:48] do you host the agents and access LMS? What is the cost model? Um so I'll give you what's typical. So what's typical is your agent system is uh typically um
[03:30:01] what we would call containerized. So you're building you write your Python code but you execute it inside of a container which is like um something like Docker and that container which is a um kind of
[03:30:16] an isolated compute environment for that agent system is hosted in some uh deployment environment that you have usually something like a Kubernetes which manages containers and orchestrates containers. Um that's a
[03:30:31] very typical setup. Um so you write all your Python code, it it gets built into a Docker container. That Docker container gets hosted somewhere in your uh cloud infrastructure like um uh AWS or Azure, Kubernetes. Um
[03:30:48] and uh you're so usually you're paying for the infrastructure. So you're paying for the compute cost of hosting that container. Um, container. Um, and then you are typically interacting
[03:31:02] with the LLM through an API. So your agent is uh calling the model. Um, and you usually have uh an API key you're setting up as part of that agent. Always we'll see this in the demo is like the
[03:31:16] LLM we set up as part of the agent is is just really interacting with the LLM over an API call. And so you're paying over an API call. And so you're paying for the token usage there um of your API
[03:31:29] calls. And so um that's you know some fixed cost uh per token that you're paying uh that provider. Um and your but your your provider. Um and your but your your agent is very similar to any other
[03:31:44] being hosted on your own infrastructure. So you're paying for your infrastructure like your compute uh your container compute resource. Um, so
[03:31:56] then you're paying for the token usage. You're usually paying for those two You're usually paying for those two things. um examples of that LLM interactions. We'll see an example of setting that up
[03:32:11] We'll see an example of setting that up as part of our agent setup. Um, but uh we won't be going into like the deployment aspect of this stuff because that's a little beyond the scope of this. that's more for like a engineering
[03:32:25] this. that's more for like a engineering uh DevOps kind of setup.
[03:32:37] further questions on that or did that help answer a little bit? infrastructure usually like a Kubernetes cluster that you have managed
[03:32:52] in your cloud like your Google GCP or Azure or AWS Azure or AWS and you're uh hosting your agent code which is going to be basically like a Python system like you would any other
[03:33:07] software basically exactly the way the same Okay.
[03:33:27] I wanted to move into a little bit more on crewi in particular the uh core elements of it that we would use to build a multi- aent system. Um and so
[03:33:40] the four things that we need to talk about in order to uh build that up are going to be agents, tools, tasks, and process. Um so uh just at a high level,
[03:33:52] the agents um are going to be objects from the uh framework that we will obviously utilize to do things. So we will be able to build agents relatively will be able to build agents relatively easily from Crew AI. Um and then the
[03:34:06] easily from Crew AI. Um and then the tools are what the agents use to do things right to take action. So these are things like um calling an API like doing an internet search um executing a custom function maybe even calling a
[03:34:20] model like you just suggested. Um those would all be examples of tools. So um and again in the demo what we're going to see is we have a way to build a custom tool that the agent can utilize. Um so we'll have a way to build agents
[03:34:35] from Curi. We'll have a way to build tools and um assign those tools to agents so that they have access to them. Um because every agent in order to be effective really needs tools, right, to take actions. Um
[03:34:50] take actions. Um then these last two. So the tasks are um essentially like uh they they are steps that we want the agent to or the steps that we want the agent to or the agents to take um or or kind of like
[03:35:05] miniature goals to keep them on target um and kind of give the agent um uh something to focus on so that they know how to delegate. So tasks are important
[03:35:17] to assign to agents so that the delegation can kind of map the overall prompt into okay we're going to break the overall prompt down into smaller tasks and this agent has been designed to accomplish these kind of tasks. So
[03:35:31] this is a good agent for this step. Um this agent over here is a good agent for this step because it aligns pretty well with that task. Um with that task. Um so we will uh practice defining tasks
[03:35:45] which help orient the agents on what kind of work delegation they should be kind of work delegation they should be um focusing on and then process is just the way we will orchestrate the agent. So whether that's in sequence or whether
[03:35:59] that's in kind of like a manager worker setup um or maybe in like a a debate setup in a similar way to autogen and like the group chat concept um process
[03:36:11] just defines how we orchestrate the crew, how we manage the crew, how the crew, how we manage the crew, how does it get managed um so um we're going to go through and um have some examples of each one of these of core components
[03:36:25] of career. So, agents, tools, tasks, and process. Um, and and we're going to build out some demos to show off each one of these things that are going to be critical to, you know, putting together a full agentic system.
[03:36:40] a full agentic system. So, that'll be kind of our focus is breaking each one of these down into um some examples really.
[03:36:52] with tools because um it makes sense to define tools first and then of course um have those tools be utilized by agents. So uh the first place we're going to start is with tools as I said and um tools are what you know give agents
[03:37:09] tools are what you know give agents their ability to take actions right so um these could be like uh web search tools, data analysis tools, code tools, data analysis tools, code execution tools, um file writing tools,
[03:37:24] many many tools that are out there. Um the two primary sources of tools that the two primary sources of tools that that uh we are interested in when we build an agent are lang chain. So this is a library that has um is a very
[03:37:41] special library that helps us build applications but also helps us build applications but also helps us build custom tools for agents to use. Um and I'm going to show you how we do that in code shortly. Um so we can build uh lang
[03:37:56] code shortly. Um so we can build uh lang chainbased tools very easily. Um KU also has a number of tools built into the library that you can pull off the shelf. Kind of like how in you know um uh Caris from uh TensorFlow we were able to pull
[03:38:11] from uh TensorFlow we were able to pull in models off the shelf. Um same idea tools off the shelf as well that are already pre-built that we can assign to agents. Um the the thing that I want to show us in
[03:38:25] the demo is how do we build a custom tool to do something we want to do and a very good example you mentioned earlier is something like maybe uh run a prediction through a model. Um that could be a custom tool. Um I want to
[03:38:39] show you how lang chain helps facilitate building custom tools. um which could be we could think of them as just basically functions, right? Tools are just the way for the agents to do things like edit a file, do a web search, um do a database
[03:38:56] query, um integrate with Slack, integrate with Google Sheets, integrate integrate with Google Sheets, integrate with um Salesforce, whatever it is. So with um Salesforce, whatever it is. So um so one of our our um ideas will be to
[03:39:11] examine how we can build a custom tool and then make that tool available for the agent to use when we define our agents. Um so uh but as we said earlier you know
[03:39:25] so uh but as we said earlier you know tools are a um hot topic because they there's a lot of integrations out there that um can be difficult to work with and so um we're not going to get into it here but
[03:39:40] one of the things that makes uh connecting agents to tools very possible is something called MCP. We kind of highlighted this um you know last week but um we we won't need that here because um the tools that we will build
[03:39:56] because um the tools that we will build will be uh um entirely self-contained to the library um and and entirely used within the crew AI ecosystem. So, um, you know, MCP would be more
[03:40:09] relevant for something that exists outside of that ecosystem that we may Salesforce integration or something that Ceri would not already have. Um, and then and in which case we need a way to communicate with those tools and use
[03:40:24] them. Uh, which would be more of the realm of MCP. But these are going to be tools that are entirely, you know, Python based, entirely built uh with or in the Crew AI ecosystem that we can use um with our
[03:40:39] ecosystem that we can use um with our agents.
[03:40:53] Okay. So some of the uh key features of tools So some of the uh key features of tools that we are really looking for is um that we are really looking for is um versatility. So basically uh we want the
[03:41:05] ability to have um pretty common tools that are that any agent would want to use. So things like the web search we talked about um maybe data analysis, talked about um maybe data analysis, maybe uh code writing or file editing.
[03:41:19] maybe uh code writing or file editing. Um so so we want a good range of tools and luckily that's doable. Um whether that's from the toolkit that creati already provides or we can go out and write our own very simply using lang
[03:41:34] chain. Um so I'll show us how to do that. So we want um versatility and again we get that not only versatility but customization which we get from you
[03:41:46] know being able to build our own our own tools. integration. So, we want uh agents to be able to utilize these tools um without any issues. And so again, everything that we will um look at will be entirely
[03:42:00] that we will um look at will be entirely within the Crew AI library and package. Um and so that really won't be an issue if we had to use tools that were again outside of that um that were integrating with some custom software like a uh a
[03:42:18] may need something like to integrate that. We may need something like an MCP, but we won't need that here in the examples we're going to do. Um customization. So, we want the ability to uh create new tools whenever
[03:42:33] ability to uh create new tools whenever we need them. Um and so, uh we will have that ability um fairly easily again using a a library like Langchain and Korea is very compatible with Langchain um tools. So, so that's very nice. It
[03:42:49] link chain provides a really nice um template for building a tool and then query kind of understands that template so it can work with those um link chain so it can work with those um link chain built tools really easily.
[03:43:06] Um we also want built-in error handling. So uh like we want nice logging, we want nice uh exception handling where necessary. So luckily the tools again necessary. So luckily the tools again that are from offtheshelf from Crew AI
[03:43:19] um are built with those in mind. So they have very nice uh logging and kind of uh we built our own that's something we may consider adding um is how to handle any
[03:43:33] tool usage errors uh and what to do as kind of a fallback or a default. Um but generally all have that. So that's very nice. It's very easy to use. Um, and also, uh, working with tools inside
[03:43:51] of Core AI, um, we try to minimize, uh, redundant operations as much as possible. So, we get efficiency. Um, that just means that, uh, you know, as we're using tools and we generate, uh, context from that, like we get a result
[03:44:05] from that, um, we have the ability to, um, uh, career can internally cache um, uh, career can internally cache that. So that um you know we don't need to redo that operation if we're trying to to grab that context. We can kind of
[03:44:19] just grab it from memory. We don't need to go back and do another tool usage to go back and do another tool usage just to grab that result. Um so that's that's possible and that goes back to that shared like state
[03:44:31] back to that shared like state management we talked about earlier. want to take us through on building tools. Um
[03:44:45] so I will uh share that document with you in the chat and then I will um show you some of the code to build a couple tools. The the tools that we're going to build are going to be used as part of our overall agent system that we're
[03:44:59] tools that we're going to build are going to be an internet search tool going to be an internet search tool mainly for the travel researcher to be about different destinations that we could travel to. And then a calculator
[03:45:14] tool which would be useful for doing like budget calculations. Um so I have uh both of these tools so I have uh both of these tools available to our agents um you know if
[03:45:26] they need to use them. So I want to show us how to build those um custom tools uh us how to build those um custom tools uh using um lang chain. So let me go over to that and then share that demo with you.
[03:45:41] So again these are some demos that are included in our AGS course. Um let me go and find that one.
[03:46:03] And then let Okay, I got it. Let me share it with you.
[03:46:15] Okay, so I uploaded that and then let me share my screen.
[03:46:27] obviously you wouldn't have had this before because again it's not part of our regular materials, but just providing it here as again it's part of providing it here as again it's part of our AGS course or AGS program um on
[03:46:41] specialization. Um, but just walking through this here. Our goal in this demo is to see in the code how we can build some custom tools that our agents will use. So that's that's
[03:46:56] can build custom tools for our agents to use. Um, and so what we're going to do use. Um, and so what we're going to do is um I'm going to show you how to set is um I'm going to show you how to set up the uh example project in VS Code.
[03:47:11] Um, this this demo uses VS Code. It does not use collab, although you could get away with doing a lot of this in collab. That wouldn't be an issue. Um, but it assumes you're going to write uh py files and rather than do everything in
[03:47:26] notebooks. Um, so I'll show you that setup when we get into the demo. Um, but essentially we're going to create two files, a calculator tools.py file and a
[03:47:38] search tools.py pi file and we're going to put some code in there. Um, so so we're going to have uh those two files and we're going to put some code in and we're going to put some code in there uh that I'll show you um to end up
[03:47:53] making the custom tools. Um the thing that we're going to do, I'll show you the code for it in a moment. Um, but the thing that we're
[03:48:05] going to take a look at that helps all of this work is really this lang chain of this work is really this lang chain tool decorator here. So, this thing um is immensely help helpful for building a custom tool. This uh we haven't seen it
[03:48:21] yet in our Python code, but this at tool is what's called a decorator is what's called a decorator um in Python. what's called a decorator um which is kind of like a special way of um inheriting the um features of that
[03:48:39] uh decorator. Essentially what this does is it makes this function behave like a a standard tool template. You think of the tool as kind of like the at tool as providing a template. Um uh as kind of providing a template and
[03:48:57] um this uh this decorator is kind of dictating that this function should look and behave like a lang chain tool. Um
[03:49:13] share it again. and it was right above your message.
[03:49:27] I just I just posted it again, so hopefully you have it now.
[03:49:41] to I'm going to open up this code in VS Code in a moment and share the code file Code in a moment and share the code file with you so you have it. Um, but uh essentially um the demo wants you to write out this file, this calculator
[03:49:55] tools file. The main feature of it is this tool decorator and because this is like any other Python function essentially. essentially. But what it when you have this at tool,
[03:50:08] it makes this function um behave like a tool from Langchain. It makes it act and behave as if it was a tool from from Lang Chain. And why
[03:50:20] tool from from Lang Chain. And why that's so helpful is because um crew AI agents know how to work with these tools very well. They know how to work with it seamlessly. And so um when you instantiate an agent and you work
[03:50:35] with the agents that which is what we're going to do in later demos um they will be able to work with these tools um very very easily because this thing has kind of a stam standard template to it. It's this tool decorator. It it makes this
[03:50:50] function behave very much like any other link chain tool which is how which is link chain tool which is how which is kind of like a standard interface. Um kind of like a standard interface. Um and that's how the uh um crewi agents
[03:51:03] and that's how the uh um crewi agents know how to work with those tools. know how to work with those tools. Okay. So, um I'm going to share the code Okay. So, um I'm going to share the code with you that um that's uh in this um in
[03:51:16] a moment, but that's one tool we're going to build. The second one is inside of a search tools.py, and this is one that is going to do a and this is one that is going to do a web search um by sending a URL uh uh API
[03:51:34] request essentially. And um so the basics of the code kind of looks like basics of the code kind of looks like this. And what you can see is again the thing that's um formatting this tool to be in a standard template so that our
[03:51:47] crew agents can work with it is this tool decorator. So there it is again. It goes right decorators in general are prefaced with an at you know the the at symbol and they go right above a function right above a function. And
[03:52:03] what that does is it means this function follows this interface or it follows follows this interface or it follows this um format or this template. Um and this um format or this template. Um and so this is a internet search tool and
[03:52:16] this is our logic of what that tool's doing. And so you know the question was asked earlier could we use a model? Yeah. All we'd have to do is come in here and build write some code that um has a decorator with the at tool and
[03:52:30] then has a custom function to um process a model, right? So process some data through a model and generate a a prediction like a
[03:52:42] probability or a class you know of whatever that prediction is like if it's going through a neural net what is the class of that prediction? Um or regression like what is the value? Um but like this function could do
[03:52:55] anything. In this case it's doing an internet search which is um just calling this Google URL sending our query through in a in a request an API request um and returning the top four results and parsing those top four results. So
[03:53:11] the rest of the code it's kind of cut off here but again I'll share the file off here but again I'll share the file um is posting. So this is an an API post um is posting. So this is an an API post request to that URL with our data query.
[03:53:24] request to that URL with our data query. Um and we are parsing the results here Um and we are parsing the results here um to take a snippet of the returned result from the top four web pages um that are returned. And um that's going
[03:53:38] that are returned. And um that's going to be our research uh tool. essentially do a web search, return some data from that um and uh join that together in a string. So all that gets put back together. All those top four results get
[03:53:54] together. All those top four results get joined together in this string. or less it's a function but um decorated by this tool which
[03:54:06] makes it special. That's what makes it behave like a standard tool. And this at tool comes from langchain. So this decorator comes from lang chain.
[03:54:18] You can see here we're importing it from langchain.tools. We're importing the tool decorator um which is uh used to make these function look like a standard tool that CI knows how to work
[03:54:32] with, knows what to do with. with, knows what to do with. Okay. So, let me pop over to uh so that being said, let me pop over to VS Code and um share the code file with you for
[03:54:44] the demo and then um kind of walk you through the project setup if you're through the project setup if you're interested in following the setup. Um to VS Code
[03:55:00] VS Code and share with you the files.
[03:55:13] screen? Okay. I'm over in VS Code now. Um you Yeah, you can use Collab. There there's no reason you can't use Collab. Um, you can basically put this code in a cell in Collab and it would do the same
[03:55:29] thing. Um, so certainly, yeah, you can feel free to use Collab. I'm just going to use uh VS Code and show you how you can set some of this up if you want to can set some of this up if you want to um using VS Code, but absolutely you can
[03:55:43] use Collab. If you're using Collab, just put this in its own cell. So the code that you see in AP py file could go in its own cell. That's the only thing.
[03:55:56] All right. Okay. So um a couple of things about this demo that I want to make you aware of is first of all the project structure. So what I did is I started a new directory, a new folder and you can
[03:56:12] call that folder whatever you want. I called it this MSAI demos. Um, but you can call it whatever you want. So I have a a a folder here that I created. Um,
[03:56:24] a a a folder here that I created. Um, and I um, you know, initially that and I um, you know, initially that folder was blank. But what I did is um I folder was blank. But what I did is um I inside of a terminal inside that folder
[03:56:38] inside of a terminal inside that folder um I used a tool that I uh want to uh share with you all which um we haven't used before called UV. used before called UV. So it's a it's a CLI tool um which is
[03:56:53] used for managing Python projects and installing dependencies in particular. Um, so it's called UV. Um, and so you can there's there's a few ways to install UV on your system. You can, if you have Python installed already, you
[03:57:09] you have Python installed already, you can actually just pip install UV um, and that will uh install UV so you can use it um to to manage your projects. Um the
[03:57:23] it um to to manage your projects. Um the other way to do it is to um go to this um link and follow the instructions to install. So they have Windows and Mac OS instructions. So um again it's a it's a
[03:57:40] terminal and CLI tool to manage Python projects. Um so you can follow that link there. And and by the way, this is what um essentially everybody in the industry uses to manage projects. It's super
[03:57:54] popular. Um and uh I've used um UV for a couple years now and it's it's fantastic for managing projects. Nearly everyone I know uses it. Um so I think it's a good tool to be aware of. Again, it's just
[03:58:10] it's just for managing projects, managing dependencies, virtual environments. Um especially for like your local machine. No, if you're using Collab, you don't need UV. Um if you're using Collab, um
[03:58:25] you will need to install some packages into the Collab environment. Like for instance, you want to install Langchain. So you would want to run the typical like pip install lang chain inside of a
[03:58:38] to get that to run. So something like something like that. And I can paste that in the I could paste that in the chat. You'll want to run something like that in order to install. But if you're running locally,
[03:58:52] um that's where you want to use UV. You want to use something like UV to to manage your dependencies. Um, and it's also really fast. So, it installs packages really fast. It resolves dependencies really fast. Um,
[03:59:10] really fast. So, it's a really, uh, amazing tool to kind of, um, do everything with your Python project. So, highly recommend exploring it and and maybe trying to get it to be used for
[03:59:23] these demos if you can locally. If not, Collab works fine. You just have to install. you have to run your pip installs inside your cells, install the dependencies for this. Um,
[03:59:36] Um, so once you have UV installed one way or the other, hopefully you're able to get it installed. Um, you can pip install it. You can follow the instructions on that I uh pasted earlier. Um, it is
[03:59:53] you'll know it's working if you can run the UV command. So, if you can just in inside of your terminal um you can run UV and you see all of this uh kind of menu of options, you know
[04:00:07] that it's working. Um so, you want to get to that point where you can you you can tell that it's working. And the reason that is is because basically you want to open up your
[04:00:20] terminal uh within VS Code. You want to have that blank folder available and you want to initialize um you want to initialize the project by
[04:00:32] running the command uv innit and that will kind of initialize the project inside of that folder and it will populate a bunch of these files for you. Um so I'll paste that command. So in a
[04:00:47] blank folder if you run the UV innit it will um initialize the project for you and give you a few of these files. It'll um give you a main.py. It'll give you a
[04:00:59] pi project toml. It'll give you a readme. It'll give you a Python version version you want to use. So if you want to use an older one like 311, 312, 310,
[04:01:11] you can change that in here. Um, and then your virtual environment would use that older version. Um, this is the latest, so I'm using the latest. Um, but you have flexibility to to change
[04:01:25] that. It now when you first initialize a project, it gives you a main.py that's very blank and basically just gives you a print statement. Um, and and uh,
[04:01:38] so let me make a new file just to demo let me make a new file just to demo that. So what it should do
[04:01:50] main file that basically has some code like print um hello world something along those lines. Um print uh with maybe the name of your project and maybe a little bit more code. But it gives you some kind of
[04:02:05] some basic Python code like this. And the reason it does that is so you can test and make sure everything's working. um by doing uv run and then your code. um by doing uv run and then your code. So something like main.py or in my case
[04:02:20] I I already have a main.py so fake main.py which would print out that u which will just execute that the code that's in just execute that the code that's in that python file.
[04:02:41] maybe, um, I'm going a little fast. If you're in Collab, This is just for the folks that are running locally on their own machine, showing it to you on purpose because I think it's a good industry tool to be
[04:02:57] think it's a good industry tool to be aware of in the Python world. But no, if anything UV. No, nothing you need to do is UV. You're just going to use pip to to write all your code in cells. So, collab nothing UV. UV is only for local
[04:03:14] local Python. Um, yeah. Yeah, it should be. So you want to in your main when you run UV innit
[04:03:33] folder. Um actually I can show you what that I can just do a new folder. I'll just do a blank folder and show you what that looks like.
[04:03:51] um new demos. this demo wasn't done in a notebook. It was meant to be done in VS Code locally,
[04:04:03] but you absolutely can you can paste the code into the notebook. I'll share the code in a second. I'm mainly just wanted to show you how you can set up a project in general. I just want to show that off quickly because I think it's it's useful
[04:04:15] to know. I think it's useful to know. So, what I'm going to do is open up a new blank folder. So, there's nothing in new blank folder. So, there's nothing in this folder.
[04:04:31] right? It has nothing in it. What I'm going to do is launch a new terminal going to do is launch a new terminal and I'm going to run the on. So assuming you you're local. So this is not for any of the collab users just for local. Um
[04:04:46] of the collab users just for local. Um you can run uvit which will initialize a project and you can see right away it gave me a bunch of files on the left. Right? It gave me a bunch of files on the left and um it
[04:05:01] bunch of files on the left and um it gave me this main file in particular with some basic print code. And the way I can test that my the way I can test I can test that my the way I can test this is all working is if I can run uvun
[04:05:15] main.py. What that's going to do is create a virtual environment for me and it's going to um execute this file.
[04:05:28] So it we can see it created the virtual environment. It is using this Python version because that's what's in this that's what's in here. and it executed this Python code to run to to run that, right? UV run main.py.
[04:05:43] So if you're if you're local only for local, highly recommend using uv to manage your project and install packages. Um, and I'll show you how to
[04:05:55] do that in a second for the for the the tools demo we just uh walked through. code, let me share that in the uh let me share that here.
[04:06:12] Um so the the tools themselves, what you're going to want is the uh uh let me find it.
[04:06:43] mainly you don't need to. It's just part of this demo wasn't designed to use um a notebook primarily because when you're So I'll give you the main reason. One of them is when you're working with agents um when you're working with
[04:07:00] agents generally you build a UI to build to work with agents and UIs don't display very well in a notebook. So the UI component that we're going to build for this demo and with CURI is not going
[04:07:13] to dis it's not going to render very effectively using a notebook. So generally when you're building like an agent system and you have a UI for it, it's going to be in in Python code that's going to be outside of a
[04:07:26] notebook. So I know we haven't done it for other projects, but specifically for interfacing with agents, it's very common to have most of your code in in py files and then have some of that py
[04:07:41] py files and then have some of that py code be a UI. And notebooks don't interact with UIs very well at all in Python. It's just one of those things. Python. It's just one of those things. It's hard to like launch and manage UIs
[04:07:53] from a notebook. So, and that's just because we need like um that mainly because we need uh some code to to have some way to interface with
[04:08:05] to to have some way to interface with the agent. you're totally out of luck. Um, you can still build them in uh in Collab. You Collab. The thing that you're just not going to be able to do is build the UI
[04:08:21] to interact. You'll you'll have to interact with them just through submitting a prompt in the in in the code itself, like hardcoded into the code. You won't be able to um unfortunately build a UI to interface
[04:08:34] You can build the agents, you can build the agent system. Um, so when you're submitting a prompt to the agent, you just will have to hardcode that prompt into the cell. You won't be able to launch
[04:08:50] EUI and like type in the prompt, if that makes sense. That's the only thing you'll be limited on. Everything else like building the agents, you'll still like building the agents, you'll still be able to do in Collab.
[04:09:11] So, uh give me a moment. I will share the code for you that is in those tools.
[04:09:33] so I'll share the code with you that's in those tools. So here is the calculator tools um Python file and the
[04:09:45] Python file and the uh search tools Python file. And I'm actually going to o just open this back up to the folder that I had
[04:09:57] this back up to the folder that I had um that had those tools. the code inside of that calculator tool. So again, if you are working in collab,
[04:10:11] you basically just want to copy and paste this f this code um into a cell. That's all you need to do is just copy and paste this into a cell and then you can execute that cell. Um in order to run that cell, you're
[04:10:25] going to need to install lang chain. So you're going to need to run, if you're in collab, you're going to need to pip install lang chain in order to get this to work. that's in collab. If you're in UV,
[04:10:41] the in in locally, the code that you're going to need to um the code that you're going to need to run to install anything is UV add. So UV add is the equivalent
[04:10:54] of kind of pip install. It adds packages to your virtual environment. And the package we want to do here is lang chain. So uv add lang chain. Now, I've already I already have it installed, so it's not going to show anything,
[04:11:09] it's not going to show anything, but um if you perfect. You won't need to. So, this this code should work fine where you copy and paste this into a cell and build this uh tool, this calculator
[04:11:24] tool. Um, again, if you're working locally and want to add, you want to install the lang chain package into the virtual environment, it's going to be uv lang chain. And one thing you should see is when you add packages, they will
[04:11:38] automatically show up in this pi project tol. So, it already shows up here. Um, because we haven't added them. They'll be later in the demos. Um, but at this
[04:11:51] point you should see lang chain that should show up here in this dependencies should show up here in this dependencies list automatically when you run UV add. should install really fast. It should be lightning fast install. It's really UV.
[04:12:05] That's one of its uh known amazing qualities about it is that it installs things really quickly. it it's able to resolve all your dependencies really fast, figure out which version is compatible with what else you have and
[04:12:18] compatible with what else you have and install it really fast.
[04:12:31] so UV add lang chain. If you're in collab, it's pip install lang chain, but it sounds like it's already there, so you may not even need to run it. And so you're going to want these two files or these two in collabs.
[04:12:46] Copy and paste this code into another cell. Um, but see how this tool, um, this is the code for this search tool. What it's doing is again
[04:12:59] decorating with a tool which is making it look like a lang chain tool which is it look like a lang chain tool which is what is going to be um you know helpful to crew AI. It's going to use that in our one of our later demos with the
[04:13:13] our one of our later demos with the agents. It's going to use this tool um agents. It's going to use this tool um this tool object um and uh this is just going to send a a request to this API
[04:13:26] um and then kind of parse the results and bring them all together in this uh and bring them all together in this uh string um as as what we get out of this tool. And then the calculator is doing
[04:13:39] something really simple. um it's taking a string operation and running the eval which will kind of parse the string and execute the math. So if somebody puts in
[04:13:51] a string like 150 + 25, it's going to read that in as integers and then run the math. Uh so that's what this eval function does. So it's just it's a it's a simple tool. Obviously, we can have very complex tools that do a lot of
[04:14:06] custom logic here in this function, but this is a very simple tool which even has some error handling, right? So, we're going to try to do this. If we can't parse this operation, we will um return a syntax error. We'll catch that
[04:14:21] and return a syntax error string to say, "Oh, we tried. We had some invalid syntax." So, our tool won't actually be able to be used in that situation.
[04:14:35] if you're running locally, I would recommend putting those two Python files in a folder called tools that's inside of your project. So inside of our project folder, you can make a new folder called tools. And you want to
[04:14:50] put those two files underneath of that tools directory. So, see how I have my tools directory. So, see how I have my overall folder is MSAI demos and then I I have a tools folder underneath of that with these two files. That's how I would
[04:15:05] organize it. If you're in Collab, it's just going to be in different cells. just going to be in different cells. That's fine.
[04:15:17] By the way, uh let me pause there. Anybody running locally, any luck with UV? Were you able to get it installed? Were you able to add lang chain to your project? Any luck with that?
[04:15:40] already have an Yeah. So, if you already have an environment, um you can just pip install it into that environment. That works, too.
[04:15:59] you would do in Collab is pip install. So that works totally fine.
[04:16:11] you have your environment completely isolated to this folder. It manages the virtual environment for you. Manages adding packages to you. Uh
[04:16:24] you. Manages adding packages to you. Uh adding packages to the project. I mean
[04:16:39] command not found exception uh you may need to add UV to your path. So, I would do a quick Google search on if especially if you're on Windows. Um, you especially if you're on Windows. Um, you may need to add UV to your path. So, may
[04:16:53] maybe want to search that up. I'm not too familiar. I'm on a Mac, so I Yeah, too familiar. I'm on a Mac, so I Yeah, I'm not too familiar with Windows, but I I'm not too familiar with Windows, but I think that's a common issue.
[04:17:13] cool. So this is these are our tools. So again just to recap the tools which is again just to recap the tools which is the main uh I know we got a little um uh side benefit of looking at UV which I think is you know uh helpful to see at
[04:17:27] least once because it's a really popular package manager and if you uh um you know get into working with Python you're probably going to come across UB at some probably going to come across UB at some point but um this is what you know makes
[04:17:42] the tools is this at tool tool, right? It makes it behave like a lang chain It makes it behave like a lang chain tool. Um, and so this function now behaves like a lang chain tool.
[04:18:01] execute this, nothing's going to happen. Um, in terms of UV, I So, the initial project has a main pie that should have something like this. Um it what it
[04:18:13] something like this. Um it what it actually had was uh the actual code was actually had was uh the actual code was more like this
[04:18:25] which was a print function and then down below um it had something like this. And below um it had something like this. And so if I run uh uvun uh I can execute my main.py and it just runs that print. Um, so
[04:18:41] there's nothing to really execute so far other than just executing like inside of collab. If you run those cells with this code, it'll it'll just save this uh function for later in in our later demos. So yeah, this there's no
[04:18:55] execution that we're we're running right now um specifically, but you can feel free to run these cells which will create these classes, right? It'll create this class and create this tool class
[04:19:08] in Collab. If you run those, it'll just create this class. We're not we're not doing anything with them yet. Just showing you how to build
[04:19:20] the tools. Yeah, no worries. No worries.
[04:19:32] point of this is to show off how easy it is to build custom tools. Um, which comes all for free really from this tool. Then we can just make a function that that does whatever we want it to do and it's uh compatible with um crew AI.
[04:19:47] Uh so really easy to to do that. Um and we can put our custom logic inside of here, you know, we have an internet search, we have a calculator. Um, and search, we have a calculator. Um, and that's really just all of this logic um,
[04:20:02] uh, inside of this tool. As long as we decorate it with this tool, um, you know, that makes it behave nicely and makes it a familiar interface for Crew makes it a familiar interface for Crew AI, which is nice.
[04:20:21] demo? So far, we we still have a few things to do, right? So, what we're things to do, right? So, what we're going to do is put together next, I believe, our agents that actually will use these tools. So, that'll be coming
[04:20:33] use these tools. So, that'll be coming up shortly. We will um build agents that actually use these tools uh in in our code. So, we have to figure out how to do that. But, right now, we have a couple of custom tools that those agents
[04:20:45] couple of custom tools that those agents could use in theory. Okay. Very good.
[04:21:04] tool is all of this function. This is all of our own code. So we did all of this. The only thing we're using lane chain for is this tool decorator which is like the interface, right? So this is the interface
[04:21:20] the interface um to uh call this function when needed. So um essentially this is the uh interface essentially this is the uh interface that is that crew AAI understands is if
[04:21:34] we make this function act like a tool that's all that this it's it's almost like an inheritance of a class right it makes this function behave like a lang chain tool but really this function is our custom logic so we created the tool
[04:21:49] we just all of this custom code so this this code could be whatever we want it like you like the example you gave earlier is call a model or make a prediction that could be inside this function. We this can be whatever we
[04:22:01] want it to be but the point is it behaves like a lane chain tool which is important from a function interface perspective of crew AI. It
[04:22:14] interface perspective of crew AI. It needs it it it needs to act like a lane chain tool for it to be usable from crew AI. But this function is all of our own custom logic.
[04:22:39] No, no problem at all. It's a good question. Okay. So, we have that. We have these two tools. Um, and we're going to, as I
[04:22:53] said, so where we're going next is to build a a couple agents that will use these tools and then ultimately put those agents together with our um uh
[04:23:05] those agents together with our um uh crew to to um have our our multi- aent crew to to um have our our multi- aent system for doing an itinerary planning and travel, you know, agent um system. So, uh, that's what we're building up to
[04:23:18] and and so our agent will have access to these two tools and we'll see how we can these two tools and we'll see how we can make them utilize those tools um, you know, as we uh continue building out the rest of the demo. Thank you. Um, so
[04:23:33] yeah, we can we can continue here. So, I'm going back to the notes. Um, we saw I'm going back to the notes. Um, we saw the first uh example of building a tool. I um want to go to uh now building out the
[04:23:47] agents as part of the crew which is kind of the main component of it all um that makes everything work of course is the agents. Um so inside of crewi this is going to be our main building block for um kind of
[04:24:04] an object that represents an agent in the code. So, and this is what um we're really interested in because this is the thing that interacts with the LLM. It utilizes tools to try to accomplish tasks, right? Um
[04:24:20] and so, uh for example, you could have a customer support agent focusing on resolving issues, a sales agent, um generating leads, securing deals, on and on and on. every agent can have a unique role and that's actually going to be
[04:24:34] crucial in setting up the agent as far as an object goes. So typically when you're using careeri that these are the building blocks you're you're you may define custom tools like we just did but then we have to define agents
[04:24:48] which will utilize those tools but also utilize LLMs to complete tasks right um utilize LLMs to complete tasks right um so they they're obviously a huge huge um building block of building out the the system and so we need to learn how to
[04:25:04] build agents and then see some examples of that inside of the uh crew AI uh library. So in order to build out our team or our
[04:25:16] So in order to build out our team or our crew of agents um we generally want to crew of agents um we generally want to um you know know what our end goal is. So what is the overall uh task that the agent should be accomplishing and
[04:25:31] typically what kind of tools will they need to get that done. So, every agent need to get that done. So, every agent that we define will have these kind of core elements. They'll have um tools at their disposal. They'll have an LLM they
[04:25:46] can interact with, but they'll generally also have um kind of task guidance of what is what should this agent focus on or what is it good at? What should be or what is it good at? What should be its goal? Um, and so, uh, for each agent
[04:26:00] generally want those elements. And we're going to see how to initialize those as far as the code goes. But, but when we build our team, we should have an overall kind of goal in mind of what the workflow is. What are we trying to
[04:26:14] generally accomplish with this agent um, system? What kind of queries is it typically going to be answering? Um, and then that can help us define which agents do we need to do that. Um, and it and you
[04:26:28] know, it may be that uh we might only need one agent to do that. Maybe we need a broader team of agents because there's going to be lots of smaller tasks that these agents handle. Um, that might all be different from system to system. It
[04:26:42] just depends on how we decide to break those down. those down. So and then when we build our team and our team has those agents um we then have the ability to uh set up how they
[04:26:57] will execute. So um I know it says here appoint a leader um and that is possible. So within Korei you can set up a kind of manager worker setup and we're going to explore that in the processes um section but um you don't technically
[04:27:14] have to. you can have all the agents kind of collaborate together without a kind of collaborate together without a single manager worker setup. Um but it is very common to basically set up and appoint one of the agents that we build
[04:27:27] as kind of the manager and then um that one will be responsible for doing a lot of the planning and then delegation to the other agents as workers. Um, that's
[04:27:39] uh um still, you know, uh that could be something you could do. And I'll show you how how that kind of looks inside of you how how that kind of looks inside of the code um to appoint a leader. We
[04:27:54] don't necessarily need to, but um you know it we could, and I'll show us how. But what's bigger than than necessarily a manager agent is having roles. So each
[04:28:07] agent that we build will have a specialized role which will align it to specialized role which will align it to particular tasks. Um so you know whenever we're delegating work um a you know agents will have roles that will
[04:28:23] align them to those subtasks uh so that they can tackle those um and they can kind of act as specialists to accomplish those. And it could be, as I said agent that has access to very specialized tools and, you know, a
[04:28:40] special LLM. Um, we can have we have the freedom to set all of that up and we'll see that uh as we start building our agents. So, we'll we'll define a role. Um, that'll be a really big part of defining the
[04:28:54] be a really big part of defining the agent along with the tools in the LM. agent along with the tools in the LM. The role will be really critical, too. So, uh, when we define an agent in Curi, these are all the things that we should
[04:29:07] be filling out for that agent and we're going to see that in our code. The these going to see that in our code. The these are all parameters of the agent object that we should be filling out in our code. Um, so they kind of align to what
[04:29:20] I just talked about. We have a role which is a big one that basically specifies the expertise of this agent with as part of the team. So what should
[04:29:32] with as part of the team. So what should this agent focus on? What is its primary role? What is its expertise? Um it could it doesn't have to be it's a string but string. It could be a short and succinct um string almost like a job title. So
[04:29:48] this like for example we can have an agent that is a research analyst. We can have another agent that is a writer. Another agent that is a um Another agent that is a um uh a reviewer. Um so they can be very
[04:30:03] short titles but they kind of provide like a general role of that agent within the team which helps when breaking down work and and you know assigning that to various agents. So the role is a part of that and think of the role almost like
[04:30:19] You sometimes you assign a role to LM and say you are a research analyst doing this this this and um that helps kind of orient the LLM to its tone and audience
[04:30:32] and etc. It's sort of the same thing with the agent. The role helps orient the agent on what kind of task it will specialize in. So not only the role but these really these three things work together to give
[04:30:47] these three things work together to give background context about the agent and what kind of task this agent should be focusing on when it's when we're focusing on when it's when we're executing um an overall task. Um and so
[04:31:01] we already talked about the role it specifies the agents kind of specialty or expertise within the crew. um we further can describe what its primary further can describe what its primary objective that um directs its decision
[04:31:15] makingaking. So this the goal of which is a string again in when we define this inside of the agent object this is going to be a string and this helps uh again with delegation. So these couple of things work together
[04:31:32] to define okay what is this agent's primary goal? What is it good at? What does it typically do? This agent helps compare smartphones. This this agent compare smartphones. This this agent helps does research on XYZ. This agent's
[04:31:46] helps does research on XYZ. This agent's goal is to um write up our itinerary and goal is to um write up our itinerary and and from our planning um etc. So we give the agent kind of a general direction. This is what you this is the kind of
[04:31:58] things you should be working on generally is doing like this market research or whatever it is. So that's a string. Again, doesn't have to be very long, but it's just a description and context for what this agent's primary
[04:32:12] context for what this agent's primary objective is um as part of the team. Right? So we'll see an example of this um because we're going to be defining multiple agents in our code. So we'll see some uh various examples of what
[04:32:25] see some uh various examples of what good goal descriptions look like uh when good goal descriptions look like uh when we um write them up in our code. Finally, we have a backstory which provides kind of a personality or um
[04:32:39] context to the agent. Um and again, these three things, the role, the goal, and the backstory work together to orient this agent as part of its like its objective, its role. Um what is it good at? How does it conversate um as
[04:32:56] part of the team? So, for instance, the backstory here is just a little bit of tech journalist with a passion for reviews. Um, and so this is kind of like an LLMs when you give a a role to an LLM. Like I said
[04:33:11] earlier, uh, it's sort of the same thing. Um, you know, what is the background of this agent? And it and again, this is something that helps orient it to what is it good at? What what types of tasks should it be taking
[04:33:24] on? a former tech journalist is probably going to be good at tasks like writing um doing background research those kind of things. Okay, so these three things uh we will define they're all strings they're all
[04:33:39] string descriptions that we will set up in our code when we build an agent. The other two things are really really critical here which is defining our critical here which is defining our tools which is just a list of tool uh um
[04:33:51] objects. So um this is we built two tools already. Uh so we can basically whenever we define an agent we can put those we can reference those tools here as um basically giving access to to for that
[04:34:07] agent to those tools. So um again these can be things that come from the crew AI offtheshelf the toolkit. They could be custom tools like the ones we just built but we'll be able to grant access to those tools. um to
[04:34:23] this agent by including them in this tools list uh for the agent that we build. So every agent um we should be specifying which tools they can use. Not only that, but we should be specifying which LLM they're going to use. Now this
[04:34:39] is again these two things like the tools in the LLM give us so much flexibility in identifying what this agent is capable of, right? because depending on the tools it has
[04:34:54] access to and depending on the model that it's using to plan and think and that it's using to plan and think and conversate um that agent can be highly highly specialized, right? Because this LLM could be like one of those special
[04:35:07] LLM could be like one of those special GPTs. It could be a fine-tuned model. So, it could be like a really fine-tuned research uh model like the consensus GPT research uh model like the consensus GPT that we looked at um last week. Um, and
[04:35:22] so that may make it really really specialized in that kind of task and it has access to an LLM really geared towards that task and that's been fine-tuned for that task. Um, and every agent we can change which LLM
[04:35:36] agent we can change which LLM uh, you know, which LLM uh, it could Same thing with the tools, right? Every agent we define can have access to different tools. They don't all need to have the same tools. They don't all need
[04:35:49] to have the same LLM. [snorts] So those so these make sense like every agent will have a tool list that we can define and an LLM will be able to pass into the uh agent and then it has sort
[04:36:04] of these roles, goals and backstories to orient on what it what this agent should be utilized for. What kind of task should it be utilized for? What is its background? what is its role as part of the team? Um, and that way we can
[04:36:19] delegate effectively within the team and it can conversate figure out who should it can conversate figure out who should tackle what at and when we do planning, who should do what. Um, a lot of that is guided by the role, goal and backstory.
[04:36:31] But not only that, but like what tools does this uh or sorry this agent have access to. It's also important. It kind of defines the capabilities of the agent. So again, these are all things that we
[04:36:46] have to set up in the code and I'm going to show you that uh coming up uh to show you that uh coming up uh shortly. should be. Uh feel free to use it to ask questions or or send me a question. It
[04:37:01] questions or or send me a question. It should still be working. projects, um, yeah, definitely, uh, ask any questions you have. Slack's still
[04:37:13] any questions you have. Slack's still working.
[04:37:26] before we jump into the code ourselves, um, here is an example of building a um, here is an example of building a travel agent. Um, you can see the main import here is from Crew AI. So, we're using the Crew AI Python package. We're
[04:37:41] importing a generic agent. So, so that's our agent object. And you can see we're defining all those things we just talked about inside of this agent when we initialize it. So, we're defining a role, we're defining the backstory,
[04:37:55] we're defining the goal, defining the tools, we're defining the LLM, and we're also turning on this, which is pretty standard to turn on, which is logging visibility just to make sure every
[04:38:08] interaction with the LLM is logged. So, we see the output of that, like what are we calling the model with, what are we getting in response uh from the LLM? getting in response uh from the LLM? Uh and then um you can see the tools.
[04:38:22] These are the tools we defined earlier. This will be an example of we're using this function which is the search internet tool. We're using this function which is the calculate tool. Those are the functions that we defined which had
[04:38:35] the functions that we defined which had the tool decorator. That's so in this tool list it's going to recognize these as valid tools because they are inherited from the tool decorator. Right? So it they look like any other
[04:38:49] tool and so they're valid to go in this list, right? This list of tools here list, right? This list of tools here that this agent has access to.
[04:39:02] we're going to build out some agents. Uh and one of the things we're going to see is how do we instantiate an LLM that we can use as part of the agent. We're going to see that code as well in in our uh agents file that we'll build.
[04:39:25] agent um is actually not hard. It's just defining the agent object and then all defining the agent object and then all of this all of these fields here um that kind of outline the agent. And our goal in crew AI is to build maybe a couple of
[04:39:40] these or a few of these that build out our team and then put them together in the crew, give them give them a task and then um you know uh basically execute that crew against the tasks and and then uh they can generate they can
[04:39:55] generate results by using their tools. Um that's going to be where this is all Um that's going to be where this is all headed.
[04:40:09] a few agents for our demo. So we have those tools in hand as part of the project we were uh talking about earlier. Um and we're going to build a few agents here to make use of those tools and to uh um
[04:40:26] uh collaborative collaboratively help with a trip planning workflow. Um so we are going to build a few agents. We're going to have a city selection expert, which is for determining where we should go um in terms of travel, and then a uh
[04:40:44] go um in terms of travel, and then a uh tour guide agent to um come up with an itinerary for that destination, and then an expert travel agent to um help us with budgeting and those kind of things. So we will uh um
[04:41:01] build out those three agents in our code which will all follow the same kind of object setup that we just saw. So we're going to build three agents that are all going to basically follow this kind of setup of defining the agent and then
[04:41:15] just changing out these different parameters like a ro backstory goal parameters like a ro backstory goal tools LLM.
[04:41:27] this and let me share the demo file with you. you. Um, so this would be this one.
[04:41:57] that we're going to look at um which is going to be for creating the agents. Let going to be for creating the agents. Let me share my screen.
[04:42:14] write some code that's going to build these three agents. Um and it's mainly going to follow that uh setup we just saw with the that uh setup we just saw with the object, right? So defining the um
[04:42:27] defining the uh role, the goal, backstory, the the tools, the LLM. Um those are and what we're going to do is put all of those into an agents.py file.
[04:42:41] locally. If you're in Collab, you can just put them in different cells. You can build each agent in a different cell essentially. Um, but we are, if you're this, you're going to put this code inside of an agents py, which I'll share
[04:42:58] with you, um, because I already have it built out. build. And essentially what we're going to do is build all of those agents as
[04:43:13] part of an overall. We're just going to organize it so that these like this is a choice. You don't need to do this. We're going to organize it so that these three agents are part of this class just to organize it. So all of those agents are
[04:43:25] part of a generic travel agents class. It's not necessary to do, but it helps organize our code so that all of these agents are accessible from this one agents are accessible from this one class.
[04:43:40] agent.py. Uh we're going to have these dependencies which I'm going to again share the code with you uh so that you have it. Notice a couple of those dependencies are our custom tools. So we have um a custo a couple custom tools
[04:43:55] search tool object and then our calculator tool object. So we're going calculator tool object. So we're going to utilize those in our agent. Um we have our crew AI agent we're importing here just like we saw in the example.
[04:44:09] Um, one thing that's new is this. So, this one thing that's new is this. So, this is um, basically giving us uh, an option for what LLM we want to use, which is in particular an open AI LLM. That's a
[04:44:24] choice. You can use different LLMs like you could use Gemini, you could use you could use Gemini, you could use something from Hugging Phase. Um, there are many different integrations for LLMs that we could use. This uh demo in
[04:44:38] particular assumes that we're using OpenAI LLM. Um but we could use uh whatever one we wanted to. Um and so that's completely interchangeable, but
[04:44:50] this will be an example of using uh OpenAI in particular um as we build [clears throat] out our agent. Of course, it's going to need LM access. In course, it's going to need LM access. In this case, an OpenAI LM.
[04:45:06] moment. You can see that we're building a travel agents class and it's going to contain all of our agents are within this class and that's purely for organization purposes. There's no necessity to do that but it's
[04:45:23] just to organize our code so that those are a part of this one class so we can reference that later. Um but is it strictly needed? No, not really. this strictly needed? No, not really. this for organization.
[04:45:45] question. Are these GPT models access accessible? They used to be um uh but I don't think they'll work now because I think these API keys are expired. Um,
[04:45:58] think these API keys are expired. Um, but uh I'll show you where you can swap that out to a different model um like Google or hugging face. And so you can Google or hugging face. And so you can um utilize like a free API uh to call or
[04:46:12] if you have your own like open AI API key, you could swap that in here and use that. Um but unfortunately, no, you would need but unfortunately, no, you would need the Open AI key uh to do it.
[04:46:46] Uh, can you use local LLM? Yeah, you can. Um, it's possible.
[04:47:02] LLM part and we'll see how we can swap that out for different LLMs. Um, but what I want to get to is the agent part. So, as part of the class, we are defining three agents. One is going to be an expert travel agent which looks
[04:47:17] like this. So notice that we are defining an agent here um which is the defining an agent here um which is the crew AI agent and what we're doing is um various parameters right so we have our role which is expert travel agent we
[04:47:33] have our backstory which gets a little cut off here but it's just some some seasoned expert in travel planning and logistics decades of experience blah blah blah blah blah right and you'll see the full string in the code when we when
[04:47:47] we uh go to it. Um we have its goal which is plan a 7-day trip with daybyday details including a budget, packing tips and etc. Um you can see which tools it
[04:47:59] has access to. So we're only giving it access to these two tools but again like we could give it access to a broader range of tools if we wanted to. Um but that's our opportunity to put in the tool list what tools we actually want to
[04:48:14] have. In this case, we just have a internet search and calculate. Uh we're turning verbose on uh so that we get um good logging. You generally care about seeing all those interactions, but generally you do,
[04:48:30] especially when you're debugging. Um and then we're setting our LLM to be one of those LLMs that we defined earlier in the class. So um in this case, the OpenAI GPT. Um, but of course that that
[04:48:44] just could reference any of those models we put in um earlier we put in um earlier um as part of the uh setup. So again, I'll point that back out in the code uh when we get to it.
[04:49:01] alternate. The reason I brought up Gemini is because um you can get a free Gemini is because um you can get a free API key for Google by going to uh Google AI Studio. If you search the Google AI studio, you can create a free API key
[04:49:16] studio, you can create a free API key there and use it with one of the Gemini there and use it with one of the Gemini models for uh for this LM. Um you can also use hugging face if you you know generate a token on hugging face. You
[04:49:29] can use one of the free models there too. You can even download it locally um too. You can even download it locally um and and use it locally. Um yeah, anthropic. Yeah, you can use different uh if as long as you have an API key. Um
[04:49:44] yes. So basically all so let me go back to um there are variations on this LLM that come from different uh lang chain providers. So
[04:49:57] different uh lang chain providers. So see how we have lang chain open AI see how we have lang chain open AI um you know you can use uh lang chain uh um you know you can use uh lang chain uh Google genai or lang chain anthropic
[04:50:10] there's various uh models you can grab from the different uh integrations with from the different uh integrations with different models there
[04:50:28] you can do uh probably the best option is for free is probably hugging face. is for free is probably hugging face. Probably the best option is to use like a hugging face model here. Um which would involve uh swapping out
[04:50:43] this. So in the code that would involve swapping out this um this chat openai swapping out this um this chat openai for like a hugging face hub uh model and then like downloading the model locally to be in memory. Um probably be the best
[04:50:58] option because you could use it for free. Um so I'll show you where to do that uh once I open up the code. I'll show you where to do that. But yeah, it's basically here. We could swap it out um
[04:51:12] basically here. We could swap it out um for something from Hugging Face. I think free would probably be that and the Google Gemini Google Gemini um because you can get a free um API key
[04:51:25] um because you can get a free um API key from them and use use Google models.
[04:51:39] Uh it it depends on the model. Like I think a very cheap like an older uh I say cheap but like very lightweight model I think would be fine. So like one model I think would be fine. So like one of the older Qins might work. Um
[04:51:51] of the older Qins might work. Um that is probably fine. Might take it might take some like playing around with to get correct but playing around with to get correct but yeah
[04:52:06] would require anything like to to use one of the larger like newer ones probably it would require a better hardware but you might be fine using an hardware but you might be fine using an older one
[04:52:21] Google that's just calling the API so it's not really going to cost you you because you're just calling the API but uh there is a limit um it is free but you'll hit a limit eventually if you're using it a
[04:52:35] But you you can get an API key for free and kind of use it um through the again if you go to Google AI Studio, you can get a free uh API key Studio, you can get a free uh API key from Google to use.
[04:52:56] By the way, this is the uh travel agent. So in a similar way to this um this so similar way to that we can build the city selection expert uh which is looking at building travel destinations. Now something interesting about this is
[04:53:12] Now something interesting about this is because it's not doing any budgeting or anything. It's just selecting a place to go. The only tool it has access to is the internet search. it actually doesn't need the calculator object because it's
[04:53:26] not going to be doing any kind of budget calculations or anything. So, it only gets that tool, right? So, that's a way for us to limit what tools it actually has access to. And this one doesn't need access to
[04:53:39] anything beyond just internet search for picking the travel destination. So we have an agent with this city selection expert with a um pro at
[04:53:51] finding destinations and its goal is finding top cities to visit considering weather, seasons, budget which would be information we would have um from you know uh we would already have from the prompt usually from the information
[04:54:12] the other setups we've had. So, we're share this code with you in a moment, but we're going to have an agent that has this role, backstory, goal, uh, again, just the internet search tool. It
[04:54:25] doesn't need anything else, and then an LLM. You can see like just going back to this call out of when we define this agent that's our opportunity to you know again specify
[04:54:39] what specialty tools or LLMs we want these specialty tools or LLMs we want these agents to have. In this scenario we're basically giving it the same LLM to use. um we're not really specializing
[04:54:53] it to use a special LM, but we would have the opportunity to do that if we wanted to, right? We could we could change that LM there in that argument. Um same thing with the tools. Every agent can have different tool access. Uh
[04:55:07] in this situation, this one only has internet search. Um but you know, it could have lots of different tools uh if we needed it to. we needed it to. All right. So, let me share the code
[04:55:21] file with you that has all of this. So, this would be uh again in our agents.py.
[04:55:33] of pop over to the VS Code to show you that.
[04:55:46] So, here is our agents.py. Now, if you're um running in Collab, Now, if you're um running in Collab, you're going to need to pip install um Crew AI uh and Lang Chain
[04:56:08] So, you'd want to run those two in order to get access to uh access to these two things. So to have the LLM access and to have the crew AI access, uh you would want to run that pip install in your notebook. Um or locally you would do the
[04:56:23] notebook. Um or locally you would do the UV add if you're using UV um or even pip UV add if you're using UV um or even pip um you would do UV add uh crew AI and um you would do UV add uh crew AI and lang chain open
[04:56:35] AI. So you want to run that command which would add those packages to your virtual environment. Again that's if you are using UV locally. Um for me it's
[04:56:47] going to see that I already have those. And what you should see is those should pop up here in your dependencies list. Um so we have those two guys added for Um so we have those two guys added for this uh agents file.
[04:57:09] want to have uh those installed. And then if you're in Collab, you basically just want to copy all that code and put it in a cell and run it. Um that would be fine. Just copy all this code and you know put it in a cell which is just
[04:57:22] know put it in a cell which is just basically the class to contain all of our agents. So this is going to contain all of our uh agents that we're all of our uh agents that we're building. Um, you can see the backstory
[04:57:34] is a bit of a a string here. I'm a seasoned uh expert in travel planning, here to help plan your perfect trip. Then plan a 7-day trip with day-by-day details, including a budget, packing tips, and safety recommendations.
[04:57:48] And it has two tools. Um, it has the search internet tool we just built and the calculator tools that we just built. Um, and it has access to this LLM.
[04:58:00] Um, and it has access to this LLM. So again, if you wanted to uh change the model, um this is where we could do it. Uh basically, we could set up a new Uh basically, we could set up a new model. Um you know, depending on what we
[04:58:15] use, we may have to in uh pip install different extensions. So if we wanted to use like hugging face um we may need to install especially locally like um different uh lang chain hugging face
[04:58:29] different uh lang chain hugging face tools to get it to work with that. Um but essentially what we could do is um have a uh new model like uh Gemini.
[04:58:41] have a uh new model like uh Gemini. Um and uh we just need to uh we could pass in our um model version here like 2.5 flash or something. Um, and then what we need is to pass in
[04:58:56] the API key, um, which we would need to set inside of here, uh, somehow. So, you would have to put in your your put in your your API key here, whatever that is. Um, you
[04:59:11] know, we could, uh, set that. So, if you're using Google, you can use an you're using Google, you can use an example like this. Um, and you could just set, uh, we don't even need temperature. don't really want to change
[04:59:23] that. Um, but this is the minimum is just specifying which model we're using and then specifying an API key. Again, you can create an API key by going to,
[04:59:35] let me make a comment here. You can go to um Google um uh uh AI uh or I should say AI Studio.
[04:59:51] google.com to create a free Gemini API key. So that's just an example. Um you can go there to create a free key and then you
[05:00:04] just would put that key here and then you can use uh you can use um the Google >> And that brings us to the end of this Microsoft AI engineer capstone session. In this video we explored three Capstone projects. autonomous driving, tourism
[05:00:20] recommendation and sales forecasting. We also reviewed the important machine learning, deep learning, computer vision recommendation systems and data analysis concepts needed to complete these projects successfully. We also looked at
[05:00:32] agentic AI and understood how frameworks like Langap, Autojet and QAI can help us build AI systems where agents can plan, use tools, collaborate and complete the task. The key takeaway is simple. This capstone is your chance to bring
[05:00:46] everything together from data analysis and model building to computer vision, forecasting, recommendations, and AI agents. This session will help you move from learning AI concepts to building real world AI solutions. If you found
[05:00:58] this video helpful, make sure to like, share, and subscribe to SimplyLearn for machine learning, deep learning, generative AI, and the latest tech skills. Thank you for watching and see you in the next session.