Recommendation System Infrastructure Basics 1

[00:01] with shorter form content to teach you important Concepts this is a bit different than our longer form problem breakdowns or deep Dives but I'm hoping you learn something new and it helps you to grow as an engineer let us know what

[00:13] be doing more of this today I want to make a quick video to walk you through some of the architecture behind recommendation systems this isn't going to be an ml Focus video we'll get to those later but I wanted to share a few

[00:26] of the techniques used in modern systems because they end up being broadly used in designing systems that both operate at scale and don't necessarily have correct answers just to be clear I'm focusing on the software engineering

[00:38] side here if you think about recommendation systems they usually have user and you want to find all the things that we might think the user is going to want this might be YouTube videos or a buyable item on an e-commerce platform

[00:52] or social media post or whatever there's really two big challenges here first how can we know what the user wants that seems IL defined and might require know about the items what do we know about the user what about their

[01:07] relationships fundamentally this is an ml problem and I'm going to skip it for now because it's not the thrust of what we want to focus on for this video but the second item is pressing how do we make this system scale if we have

[01:19] billions of videos or trillions of posts how can we possibly find the one the user actually wants in a reasonable amount of time this seems like an especially hard needle in a Hast stack problem and it is but this isn't about

[01:31] making a machine learning models run fast there's more to it let me recommendations and then we'll move on to some solutions if you were to approach this problem you might start with a blackbox ml function which takes

[01:44] a user and a video and tells me how likely the user is to watch the video in this case my blackbox outputs a score after several milliseconds this is an important building block which will tell me that a given user will hate this

[01:58] particular cat video but love evans's Ticket Master breakdown great with this function available I can now try to make my recommendation system let's apply this function to every video in the catalog this will give us a score for

[02:10] every possible video we can then sort the scores and return the video which is most likely to be watched but wait billions of videos means I need to run through my ml blackbox billions of times this means I need a ton of time maybe

[02:24] two weeks to figure out the best video for them to watch crap that's not going to work since this is slow maybe we take it offline let's pre-calculate our when they come online we can feed them out of a cash easy no if we have

[02:39] millions of item or sorry billions of items and billions of users suddenly we need a data center the size of Texas in order to make recommendations this is untenable and it's slow I don't want to wait two weeks after I register for my

[02:52] Tik Tok videos to start streaming in okay okay you get the gist so we need to approach won't work modern recommendation systems really have three stages candidate generation ranking and reranking what we've been talking about

[03:07] so far is ranking taking a bunch of videos and applying an ml model to score them which results in an ordering for us to return to our clients but ranking possibilities as we've just seen it's just too timec consuming and resource

[03:21] just too timec consuming and resource intensive we need a better way enter candidate generation candidate generation is a simple idea we know some class of videos are much more likely to be watched than others as an example if

[03:33] I'm subscribed to Mr Beast it's likely that I'm going to watch other videos he creates similarly the most viewed videos on the platform have very broad appeal recommendation system you just take the top videos in the platform and feed

[03:48] those to users on a loop that's still how radio works today but we can do a we'll take each of these horis STS that we just talked about and call them candidate generators a candidate generator will take in some

[04:02] information about the user to generate a list of candidates or videos we think the user will want to watch we can Union together all of the candidates from each of the candidate generators then we'll take our ml function we had earlier

[04:14] which we now call a ranker to score each of these candidates since we now have order a thousand rather than a billion items this is much more tractable the scores are the ones that we want to recommend and as long as our candidate

[04:29] generators are reasonably exhaustive this approximates the much more expensive system I'll say it again for emphasis here we're taking some her istics to cut the problem down and make it more feasible we don't have to rank

[04:41] every video ever to get good results that's the essence of candidate other type of candidate generator which is worth discussing the top videos in the platform are videos from channels I've subscribe to are pretty easy to get

[04:55] we can do this with basic database queries but how can I build canidate generators for things like a video similar to the one I just liked or videos watched by users like me the answer to this is to use what are called

[05:08] embeddings or vector of numbers I'm going to simplify a bit and invoke the blackbox of ml here but let's pretend we have a function which takes in an input in this case a video and outputs an embedding or a series of numbers these

[05:22] smaller distance you could think of this like distance in 3D space here have some property it could be that we've trained our model our black box so that videos which are liked by similar users are

[05:34] clustered together or we could try to group them semantically based in their content like I've done here or we could try to find videos which irritated the same people the options are literally endless what we can see is that the cat

[05:46] video and dog video have smaller distance if you calculate the ukian distance if you calculate the ukian distance in 3D space then the Cat video and a chess match they aren't as semantically similar as the cat video

[05:59] and the dog video just like before if we want to find the smallest distance our most naive approach is to calculate the distance between our input video and every other video in the catalog this is way too slow but it's going to find the

[06:12] video that is most semantically similar or liked by similar VI users or whatever can generate embeddings for our entire catalog of videos including the new ones that are created once we have embeddings we store them in a vector database

[06:27] popular options here include commercial offer ings like pine cone or open- Source Solutions like face or in this case I want all my videos together with their embeddings in this database Vector databases are optimized with algorithms

[06:39] like hierarchical navigable small worlds which enable very fast approximate nearest neighbor sech basically they're much much faster than trying to calculate distances on each item with my database constructed I can queri it so

[06:54] for videoos similar to the one I just liked I'll grab the embedding for that video I just and query the database for the nearest neighbors of that embedding this will return to me the IDS of all the videos

[07:06] which have embeddings with small distance to my input in effect my database now answers which videos are similar to this one those are my candidates which I can feed to my ranker just like I fed the top 100 most viewed

[07:19] platform videos I can have Separate Tables or databases for each style of embedding so I can have a separate database of users to answer questions about videos liked by user similar to me it's common inside ranking and

[07:32] recommendation systems to have dozens of these indexes or candidate generators problem for the software Engineers working on these systems that's completed candidate generation and handwave pass ranking we'll get to that

[07:47] later let's talk briefly about reranking there are a lot of behaviors of recommendation systems that we want to tweak we probably want to give new to get exposure and for us to learn about them this is fundamental to the

[08:00] trade-off between exploration and exploitation we not only want to show you the video you're most likely to like but we also want to expand your horizons not recommending videos that a user is blocked or they might be sensitive to

[08:14] adjusting the full recommendation stack for each of these use cases would be really expensive if we needed special indexes that don't contain videos from creators the user is blocked that candidate generator is much less

[08:27] efficient instead in the reranking phase we're going to apply tweaks post Hawk if we need to provide the user with 10 ordered recommendations our ranking stage might produce 25 and our reranking stage can reorder or remove based on

[08:43] business rules privacy settings Etc this is actually an implementation of a very common pattern you'll see in Real World System design and it applies to search engines privacy Solutions ranking systems and more sometimes it's easier

[08:55] to design for the 99% case and have a layer to budge the final results and make them how you want rather than build a system which solves 100% of the going too long there's obviously a lot more we can get into let me know if you

[09:10] want to see more in the comments but just to recap some key points here we've covered how a multi-stage architecture in this case using candidate generation enables scalability for massive systems and makes seemingly impossible problems

[09:23] possible next we showed how you can use embeddings in Vector search to solve fuzzy problems cheaply and finally we showed how production systems are often layered with fine grained adjustments applied at the last

[09:35] level this makes problems more tractable I hope you found this video valuable and if you'd like more please subscribe otherwise until next time talk to you otherwise until next time talk to you later