System Design Interview: Designing Live Comments with a Former Meta Staff Engineer

[00:01] channel uh I'm Evan co-founder of hello interview and a former meta staff breaking down common system design interview questions together and today going to work through designing Facebook

[00:14] limited to Facebook uh it's any live streaming platform but the live comments feature so the the comments that come in under the video This is actually the second most popular question asked in meta system design interviews uh and

[00:27] it's really popular across all of the major fangs so it's a good one to know more of course I'm obligated to say it please like And subscribe it really subscribers is really motivating for us to continue to make these videos Stephan

[00:40] in 2025 particularly in the beginning of the year so uh subscribe you'll get the notification when those come out if videos aren't your thing head over to guide for this exact problem that you can read through um there's also dozens

[00:56] of others on the site we also have guided practice which is a tool that 20 others you get to draw on the Whiteboard you get personalized feedback as you go candidates really really love it so check it out um and there's tons

[01:09] of other free content on h.com that's going to help you prepare for interviews interviews with senior plus Engineers or Managers from your target company people just like me who are going to sit down for an hour with you um and kind of let

[01:22] you everything you need to to to know to interview so without further Ado let's get into it

[01:35] actual design let's take a moment to go over the road map or the framework that we're going to use throughout this video and it's the framework that I recommend interviews and so if you've watched any of our other videos you've seen me do

[01:48] free to fast forward um but for anybody who's new we're going to go through these six steps uh first off we're going to outline the requirements of the system and so this is your opportunity to ensure that you are on the same page

[02:01] that you're going to build this includes what we call the functional requirements or the core features of the system as well as the non-functional requirements or the qualities of the system after that we're going to outline

[02:14] a very basic list of core entities and so this can be thought of the nouns of your system often times this correlates almost one to one with the tables in your database but we're going to stop short of outlining the full data schema

[02:28] justification being that we just don't know it yet it's too early in our design because they're going to give us the context necessary to understand our API

[02:40] and so once we have those core entities down as I said we go into the API uh this is where we'll outline the core user-facing apis and we'll exchange those core entities um that we outlined in step two now from here data flow this

[02:56] design interviews um you'll need to to decide on the Fly whether or not it applies in this case uh it won't and so we're going to jump right into set five level design is when we head over to the Whiteboard we start drawing boxes and

[03:11] arrows in order to outline a very simple system where the goal is to satisfy our case we're not going to satisfy the non-functional yet our system might not be able to scale it might not be low latency it's very basic um but it's the

[03:26] minimum required system in order to satisfy the those core functional requirements so we'll get that down and then we'll move on to step six the Deep interesting this is where senior and particularly staff candidates uh show

[03:41] going to go one by one through those non-functional requirements that we outlined in order to expand upon our design um our original highle design um in order to make sure that we satisfy the non-functional requirements so this

[03:55] is where we'll handle the scaling the latency concerns security concerns that's the outline that we're going to follow let's do that

[04:09] requirements let's just take a really brief moment to at a high level explain probably familiar with Facebook live Facebook live is a feature within the Facebook app which allows users to live stream videos to their

[04:24] designing is what happens under that live stream this is going to be the comments that get streamed in in real time so as users are watching a video they can be posting comments and reacting to the video as it's happening

[04:37] those comments get streamed to all other users as I said in near real time and so of course this is not exclusive to Facebook this is true of Tik Tok live true of twitch YouTube live any

[04:50] streaming platform uh so the solution that we're going to have here applies to any of those awesome okay so let's get started with our our functional requirements are the core features of the system this is where you're going to

[05:04] agree with your interviewer on what you are designing now in our case our our straightforward I'm going to paste them in because it's going to be better than you guys watching me struggle through typing here in the live video but the

[05:19] core functional requirements of Facebook live comments are first most obviously users or viewers need to be able to post a comment that's crucial Fant CLC the second is that viewers can see all comments posted in near real time and so

[05:36] as new comments are coming in we're appending those comments to the bottom of the list or the log so that users can see them importantly without having to refresh their page or anything and then third is that you should be able to see

[05:49] all the comments that were posted before you joined and so maybe uh in the case of Facebook live Tik Tok live this means that you can scroll up on your feed of comments and it'll load or older comments that you can still view so

[06:02] functional requirements now I put some things below the line here these are interview like this but are going to be out of scope for us that's the ability to reply to comments and the ability to react to comments or even just react to

[06:16] the video in general and so one thing to note about this below the line thing don't feel obligated to do this in your interview my suggestion would be to use this technique of below the line or out of scope only if you're in the proc

[06:29] process of brainstorming and something comes to you that you discuss out loud and then agree with the interviewer that it's out of scope so if you're thinking requirements and you say something to the interviewer like well should users

[06:41] they say no let's consider that out of scope then that's something that you can specify is below the line basically a contract between you and your interviewer of these are the things that I'm going to focus on these things are

[06:54] not next up are those non-functional requirements and so non-functional system's qualities that are important to the user and so they can be phrased like the system should be able to or the system should be statements so for

[07:08] like Twitter it's like the system should be highly available prioritizing availability or consistency the system should be able to scale to XYZ right some things you want to talk about here cap theorem uh scalability latency

[07:21] compliance these are all the things you want to consider but crucially you don't want to just list those out the reality is that most systems nowadays need to do all those things but what you're doing in your non-functional requirements is

[07:34] calling out to your interviewer the set of qualities that are uniquely relevant things that you're going to prioritize when discussing the interview so let me paste in a list here of non-functional requirements that we can discuss so for

[07:49] non-functional requirements here are some things I want us to discuss so first off is scale at this point in the interview I would ask my interviewer about here they might ask you to estimate it or they might tell you often

[08:01] times when I'm the interviewer I'll just tell you here and in this case we want to scale to millions of concurrent videos and up to thousands of comments per second per video uh so this is going to be a lot this is going to be huge

[08:15] scale actually there's a there's a fun fact here that I think I put in the popular Facebook live video was called Chewbacca's mom funnily enough uh and it featured a mom thoroughly enjoying some plastic you know Chewbacca mask and it

[08:29] plastic you know Chewbacca mask and it been viewed over 180 million times and so that's tons of concurrent viewers we can estimate some portion of them were commenters um and that that would get us to you know these thousands of comments

[08:41] per second no problem and so the second thing that we're going to consider here is Cap theorem we've considered scale now we're going to consider cap theorem almost any system design interview it goes far as to say any system design

[08:54] interview and the question here is do we prioritize availability or consistency and so uh the way to think about this and we have a video specifically on this tradeoff but do you need strong consistency put in that

[09:09] another way is does every single right or does every single read of your system need to be reading the latest right and so in the case of our comment creation does everybody need to see the latest comment immediately when it comes in and

[09:23] the answer is clearly no like we want real time low latency broadcasting that's what we'll get to next but it's not an error if that doesn't happen right we would rather be highly available we'd rather you get comments

[09:36] than see no comments just because we can't show you the very latest comment system we're going to prioritize availability over consistency as it pertains to creating comments and thus reading those comments and then the

[09:50] around latency and so we do despite our eventual consistency want to be as quick as possible possible to broadcast new too much sense if you're watching a

[10:04] World Cup broadcast Messi scores a goal and then five minutes later people are you want to see those comments come in almost immediately and so we're going to actually quantify this and we're going

[10:17] to shoot for about 200 milliseconds another fun fact 200 milliseconds is what we as humans perceive to be real time and so if you're an interview if time 200 milliseconds is what what you're striving for anything lower than

[10:31] that frankly it doesn't matter humans won't perceive the uh the difference anyway and so last thing that I'll say about these non-functional requirements is that aim to quantify them where possible like we did here uh and always

[10:43] put them in the context of the system so availability greater than consistency me not that interesting but for comment creation okay I understand why same just low latency me of course but what specifically in our case Comet

[10:58] broadcasting and then we quantify it same comment about out of scope or below the line security and integrity are two things that are often important to live comments making sure people aren't saying bad things um it's not something

[11:10] that we're going to focus on in this particular our next steps are to define the core entities and then the API so we can do entities and then the API so we can do that now uh the reality is that these

[11:25] pretty quickly and particularly in this interview they're not overly interesting or difficult and so we should be able to get them down pretty quickly let's start to make this as easy as possible on you guys what are the core entities of the

[11:40] what are the tables that we're going to have these are all ways to think about this well most importantly we're going to have comments certainly we're also going to have a live video now the actual data model for the live video

[11:53] this is another team that will handle this but it's important to recognize as an entity because we're going to have relations to it so I'll write it down anyway and then we're of course going to have a user or a viewer so those are our

[12:07] core entities and we can then move on to our API and use those core entities to our API and use those core entities to shorthand our API a little bit uh so when you get to the API the easiest step-by-step formula here is to just go

[12:21] back to your functional requirements and go one by one through your functional requirements making sure that you have an API to satisfy each of these now this an API to satisfy each of these now this isn't 100% foolproof uh for non not non-

[12:33] something like a designer web crawler you know this doesn't make as much sense but if it's a product if it's a user-facing product as live comments is then this is a nearly fullprof way to do this and so some uh functional

[12:47] requirements may require more than one endpoint but the process Remains the requirements and create the endpoints necessary for each so the first thing is viewers can post a comment okay great we're going to use a rest endpoint and

[13:00] we're going to use a rest endpoint and we need viewers to post a comment so it's going to be a post because we're creating a new resource what is our resource name it's comments uh some of you have have picked on me in the

[13:13] you have have picked on me in the comments ironically um for not pluralizing my resource names so the reality is to me as an interviewer and to most people like who cares um but proper restful semantics say to

[13:25] pluralize your resource name so I've done that here for you um and then we posting this on and so I'm going to use a path parameter here and pass in the live video ID now why not put this in the body well you absolutely could um

[13:40] but given that it is required you cannot possibly create a comment uh without a live video ID it's common here to put it in the path parameters and so that's what I've done here now what does this return it can either return the comment

[13:52] or 200 what's the body well the body is the comment itself and notice how I'm like what is a comment at this point of course a comment is going to need to have some content uh you know if there were extensions here then maybe it would

[14:06] were extensions here then maybe it would have a um media attachments which we're abstracting this for now and it's just a comment it's the core entity that we've defined as a comment is what is the the post body there so we checked off one of

[14:20] those functional requirements let's go back up and look at what our next one is viewers can see all comments p uh posted in near real time um um so astute a minute this is going to be a websocket or some persistent connection you're

[14:36] right and if you know that this early in the interview go ahead and write it down that's fantastic I'm going to take the approach of assuming that we don't know that or even if we do we're just going to build things up simply and so the

[14:49] simple way to do this would be that we're just going to have some get we're just going to have some get endpoint we're going to have some get I don't type in front of you guys like this get comments endpoint which takes

[15:02] in again that live video ID and maybe it's just going to take in ID and maybe it's just going to take in a cursor so this is basic cursor based pagination um and we we'll be able to pass in the last comment ID that we saw

[15:16] and this is going to return to us any new comments that have come in since that last comment with some page size here maybe like 50 okay and so if you don't know about cursor based page donation fairly straightforward you're

[15:29] just going to pass in a cursor which is the ID of something in your database some Row in your database and then we're going to get everything after that and going to get the 50 comments after this and so we can just be calling this over

[15:43] and over again in order to simulate some sort of real time we'll talk about that more in the high Lev design and of course this is not the optimal solution it's not efficient we'll do something better but at the early phases of your

[15:55] candidate or something it's okay to put this down for now uh and you can just expand this is probably going to use some bidirectional communication or unidirectional communication but something persistent later on and so the

[16:10] next one uh viewers can see all comments posted before they joined well this is essentially the same thing right we want to get all the comments for a live video but instead of anything after the live comment ID we want it before the live

[16:23] comment ID and similarly we want a page size like maybe we want the 50 before this point in time and then if we want 50 more then our last comment ID would be updated to the uh last of that 50

[16:36] 50 you kind of see how that's happening recursively but the difference here is going past comments and another we're going future comments and so what we could do is we just add that direction pretty simply to our API endpoint do we

[16:50] want after or do we want before that's a way that we can that we can model way that we can that we can model that so there you go two basic rest end endpoints one to post a comment the other two to get comments I'm using

[17:04] basic query parameters here denoted by the question mark for the first variable and then the and sign uh for for the remaining variables posts gets all remaining variables posts gets all pretty straightforward thus

[17:19] requirements our core entities and our API this is kind of the general setup of the problem at this point we've agreed with our interviewer what we need to do Foundation we know the general apis that users are going to use to interface with

[17:32] our product and now we need to move on to that highle Design This is where things get fun they get interesting we start actually drawing those boxes um on the Whiteboard and so just like with the API with the highle design we are going

[17:46] to just satisfy our functional requirements and we can do that by going one by one through our apis and so if we go one by one through our apis we'll subsequently be going one by one through our functional requirements and we're

[17:58] essentially saying when this request is made what happens in our system what and what components are necessary in order to effectively fulfill that little bit quickly through this highle design the reality is the highle design

[18:13] straightforward and I want us have adequate time to go into deep um to go deep on the things that are interesting bya deep Dives um but let's talk about thing is that users want to be able to post the comment we have an API for that

[18:29] a post comments to a live video ID cool let's model that what's that going to look like for us well the first thing is that you're going to have a client I'm going to introduce an API Gateway it's not strictly necessary here uh it's a

[18:43] relationship you'll find so we don't need routing but I'm going to take advantage of the middleware and I'm going to introduce it uh I'm also going comment service is going to be responsible for com comment crud oper

[18:58] ation let me just move it down a little bit and then I need some way to persist these comments and so there is my comments theb great and so what's going

[19:10] that post request in order to create a new comment our API Gateway is going to forward that onto our comment service our comment service is going to craft and then initiate a query to our comment CB in order to create that new comment

[19:26] core entities how we weren't going to talk about the data model or the schema just yet now is the time where we do it when we get into our high level design this point I've just told my interviewer this is going to save a comment and

[19:39] naturally now I can say well what is a comment right a comment's going to have some comment ID that's going to be important it's going to have the video ID that this comment was placed on it's going to have the author ID um and then

[19:52] it's going to have the contents of the comment the text of the comment probably some created at field is going to be necess necessary and so in terms of our query pattern we're going to be querying by video ID a lot so we'll want an index

[20:04] on that we're also going to um be sorting by the created at column so going to make any decision about the type of database yet uh in your

[20:16] could talk about it now that would be totally fine I'm going to choose to talk because I think it's going to be more interesting the reality is for a simple highle design here any database is going to work

[20:29] SQL database is fine here MySQL postrest Etc key value stores document stores they're all going to get the job done um so I'll go into more detail on that a where your indexes would be what your primary Keys would be of course comment

[20:43] ID in this case so that's the thing for us to to handle but very simple client in the database and we return with our 200 check we now have viewers can post

[20:56] comments it's a simple crud service next up is that viewers can see all comments posted in near real time now of course as we alluded to earlier this is going to require some persistent connection you'd be wise to let your

[21:09] this is going to require some persistent connection but let me start with the simplest thing that we could do is that basic polling with that API that we outlined here so get comments for a live

[21:21] video ID that have happened since some given last comment ID that we have this could also be a time right different ways to handle this time might even be more natural um for this direction of the query anyway but in any case what we

[21:36] can do is then just have a simple client that is going to every two to 5 Seconds service if there are any new comments since a given comment ID if no comment

[21:48] ID is given well then there's no comments since that point right simple regardless of what database you chose so if it was a SQL database this

[22:02] is simply you know select all comments um where video ID equals the given video ID and created at is greater than the created at time of the cursor fine um

[22:14] document store or key value store this query is pretty straightforward so if you chose a SQL database then it's just select all comments where the video at time is greater than the created at time of the Cur CS or the last Red

[22:29] comment nothing to it uh similarly easy on a document store or similar awesome so again we're just expanding this basic crud service we can create things we can read things nothing to it we're going to

[22:42] use polling in order to just get new comments every couple seconds and those will be maybe not quite the near real time we're looking for yet again persistence uh connections will be important later but it's a good start

[22:54] for now the next thing that we needed to do is viewers can see all comments posted before they joined and so we had the same API endpoint we just specified a direction and it's the same concept here so when a user first joins the live

[23:07] video they have no comments they have no past comments uh or they have no kind of new comments yet we're going to probably first load up that live video alongside the last 50 comments or so from the video and in order to do this we'll use

[23:19] that same API endpoint we're just going to switch the direction and in our query than or equal to on the created at sign so now we want the last 50 comments that are less than the current comment and obviously we're not going to necessarily

[23:34] gate and so if that's the case we'll just start from the end of all existing just start from the end of all existing comments right similar concept there now one thing that I'll note just for fun um but this wouldn't necessarily be the

[23:48] most important thing in this interview but there's different ways to handle this concept of a user continuing to scroll up so you've probably done this on Instagram or on Facebook live what whatever it may be where you scroll up

[24:00] and it fetches the last 50 and you keep scrolling up and you want to just scroll want to go all the way to the beginning uh for whatever reason you you continue to scroll up and read comments and there's different ways to optimize this

[24:13] one really popular thing to do with pagination is to introduce a cache and this is going to be a cache for a specific user ID and in our case live video or just video ID so that would be the key and then it's going to be the

[24:27] comment as a list here and so when a user fetches comments past comments we might show them the last 50 but we're going to cash the last 1,000 this is going to have a really tight TTL and so maybe it just

[24:42] what we anticipate being the narrow window with which they might still be scrolling up and so now if they scroll up above that 50 instead of having to to issue the query we can just go grab it from their cash and give them the next

[24:55] 50 now what you might be thinking is it why don't we just give them more than 50 we can do that as well of course we can give them thousands uh and then let the matter at that point there's different optimizations here and this really would

[25:10] only matter for super popular videos where comments are coming in like crazy really quick in order to fetch the next the next group of thousands of comments or so another pattern that we could do again just because it's interesting and

[25:23] related to pagination here not specific only to this problem is that we can have it be such that we send the client over maybe the 500 last comments and then once the user Scrolls up to about the 400th then the client is going to know

[25:37] let me go optimistically fetch the next 500 because I have a feeling they might finish this list right finish this stack something similar to what we do in Tinder uh if you've read that breakdown or seen that video so I'm not going to

[25:50] include this because it's not the focal point here in design FB live comments if anything I would just briefly comment on that um but I wouldn't let myself get distracted because despite how simple this highle design is and it is quite

[26:02] simple basic crud service it satisfies all of our main functional requirements interview now to talk about the things that are going to be interesting which requirements around how to scale and how to ensure this low latency and eventual

[26:21] design be behind us it's time for everybody's favorite part which is the Dives we're just going to go one by one through our non-functional requirements and expand on this simple high level design in order to make sure that we met

[26:33] all of these requirements at the end we're going to be able to look back and requirements I met all my non-functional requirements I acce this interview give me the offer and Life's good right so that's our goal here uh so first off one

[26:47] level to which you're proactive in the Deep Dives is a function of your seniority and so if you're a junior or mid-level candidate this might actually the interview and it's more of a question answering so the interviewer

[27:00] might be asking you questions you know this polling approach isn't going to give us the low latency requirements we're looking for what else could you do especially if you're staff the expectation is that you identify where

[27:13] conversation towards these things and again the best way that you can do this those non-functional requirements so let's do that now um I'm actually going going to start with low latency comment broadcasting so this is the case where a

[27:29] everybody's client quickly as you remember we're using polling right now Seconds asking if there are any more comments of course this means that our latency is 2 to 5 Seconds as opposed to the 200 milliseconds that we were

[27:45] looking for so we're going to want to find a way to speed that up the key Insight is that instead of using HTTP request responses here where the client is saying are there any new comments and the comment service is responding after

[27:57] quering the database we need some sort of a persistent connection here so we need a way that we don't have to ask every single time but instead the comment service can just tell us as soon as a new comment comes around and so

[28:11] there's two popular Technologies to handle this persistent connection the handle this persistent connection the first is websockets and the second is about each of them I'm actually going to paste in some notes Here Again to

[28:25] struggle while typing starting with websockets websockets are bidirectional and so you create a connection like this and now at any point the client can say something to the comment service but the comment service can also say anything to

[28:39] and so in our use case you can imagine a new comment gets created or saved by somebody else and we just push over that websocket right so push instead of pull back to the client to say there's a new comment and we can do this quickly so

[28:53] this is great and it's a viable option but websockets also have some downsides so websockets use their own protocol they're not over HTTP or https and this weight what happens is that they initiate an HTTP handshake first then

[29:10] they need to upgrade to a websocket connection and you need to have all of places support this upgrade and so for example the client's going to go to the API Gateway try to do a handshake establish a websocket connection here

[29:23] right and then likewise it's going to go create a websocket connection here so but in the case where you have low balancers proxies firewalls or any of these things uh that don't support websockets things are going to break and

[29:37] this happens more than you would expect actually so often times if you're things are working in your test environment everything's great you put it's not working at all or it's not working for some users and this might be

[29:50] firewall that's not going to allow for these these websocket persistent connections um it could be that you have some production API Gateway or otherwise there are plenty of API gateways that supported out of the box AWS managed uh

[30:07] supported out of the box AWS managed uh API Gateway is an example um I think like engine X has configurations for it right so this is by no means a a task that cannot be achieved but it comes with overhead and it's a bit more

[30:20] complex and the main kind of interesting bit here and maybe I'll write this down oh I already have it there right so websocket is perfect for when it's bir directional there are as many reads as there are writes so the client needs to

[30:33] get as many things as it needs to send or post and so in the case of a messaging app this was the last video we did check that one out with Stefan we already but in the case of a messenger app this is perfect because there's

[30:45] between the amount of messages you're sending and the amount of messages you're receiving but that's not the case here with live comments the overwhelming a live video watching the live video and and the comments are just streaming in

[31:00] thousand or so you can have some users who are active commenters but the majority are passive they're just watching and so it's not bidirectional we can still post a comment periodically with our restful API that we have with

[31:14] this you know SL comments post request that was right here on this https that was right here on this https endpoint um but we can receive comments websockets seem to be a little bit of Overkill so what else can we do do the

[31:28] other thing that we have is that pastes it up here ssse or this stands for Server sent events send events and this is actually perfect for our use case in live comments because it's similar to

[31:41] websockets and then it it opens up this persistent connection uh but it's unidirectional and so in this case it's just the uh it's it's just the case that the comment service can push changes to the client but the client can't say

[31:56] this and so instead of having its own protocol like websockets do it's just over HTTP and https and so it uses uh it over HTTP and https and so it uses uh it uses HTTP headers in order to specify

[32:10] that hey I'm going to be sending this over over chunks so keep this open uh and it basically just looks like a long running HTTP request the other nice require anything too fancy like browsers have built in support to handle the

[32:23] here if for some reason you were to get disconnected and so so it makes it a much more lightweight and then given that we only need unir unidirectional us of course you can still post a comment you're just going to post the

[32:38] comment over the HTTP request response via our post endpoint which was this guy up here all right so hopefully that that all makes sense now not all API gateways

[32:50] similarly support ssse out of the box um so there are still challenges here um connection after predefined period of time and you need to go in there and update their configurations in order to elongate that or specify different

[33:03] that it shouldn't separ the connection after say 30 seconds or 60 seconds so by adding ssse it doesn't solve all of your infrastructural challenges but it unidirectional communication that we're looking for so let's let's tie that

[33:19] together I'm going to just by way of abstraction draw a line through here Gateway and then the API Gateway forwarding to the comment server um but I'm going to choose ssse and I would say as much in my interview I

[33:33] websockets and S as we just did and I would come to the conclusion that SSE is would come to the conclusion that SSE is best um and so when a client now creates a comment we are then going to look for all other users who are listening or

[33:47] watching that live video and we're going to push over ssse that new comment so that they get it and so you may be thinking okay but once a new comment comes in how does the comment service know which users to push this to do we

[34:01] need to update our database do we need to store this state somewhere and it's a great question but it's actually really straightforward and so all we need to do is have an in-memory mapping in this comment service and so a new s

[34:14] connection is established the first thing that can happen when this is this connection is established is that we're letting it know maybe via the headers which live video we're watching and so we're saying this SEC connection is

[34:26] associated with video id1 and we can just store that mapping in memory so look like so these connections these are just pointers to the respective connections and now if a new comment comes in for video id1 I'll look in that

[34:41] mapping in memory and say who all is connected to video one oh well I have SS connection 1 4 and 7 so let me send it to them and I can just do a for Loop and do something like connection 1. send that new comment that came in and then

[34:54] it's going to show up uh to the client for them to be able to render as needed so this is how we're going to enable that sub 200 millisecond latency because a new request is going to come in our comment service is then over ssse

[35:08] going to maintain this persistent connection that it can just push over uh that new comment uh but you might notice that this doesn't scale very well uh horizontally scale our comment service

[35:22] and now we have multiple of these and it might be the case that a new comment comes in and it lands on this server but the people who are watching this live video are on a completely different server and so this one doesn't have SS

[35:35] connections right it needs to somehow communicate with this one to send it things that we'll talk about next as we get into scaling so before we get there non-functional requirements we just talked about low latency and now we're

[35:49] talk about scale and so let's come back we had talked about what that issue is if the comment service horizontally scales then we don't know which user is connected to which service and which live video

[36:02] they're watching and so we don't know how to forward that onto them so let's move some things around I'll keep them on the Whiteboard so you guys have them later um but the first thing I'm going

[36:15] math now and you might have noticed I didn't do math up front this seems kind you're supposed to do math up front I disagree uh I find that most candidates do some estimates on scale some bandwidth they look at me they go so

[36:29] go design the same system they were going to design that's not useful math is very important though but math should inform your decisions and so my concrete suggestion to you is to do math in your interview at the point where you have a

[36:45] this case the decision that I want to make is scaling here and so I want to that we were going to have a million so 1 two 3 one two three we said that we were going to have a million uh active

[36:58] live streams at any given moment and then we would have like a th000 viewers we said right and so what I'm trying to figure out is how many ssse connections because an ssse connection is probably going to be my limiting factor here uh

[37:14] boxes high powered boxes they can handle give or take like a million ssse connections and so we'll use that as a rough number and then our division million concurrent videos a th000 viewers of each and so that means that

[37:28] we're going to need a thousand services in order to handle just the number of ss connections and so we don't probably need a th Services when it comes to writing posts um because the amount of memory that we have here the network

[37:42] scale this out later it would turn out that that's probably less and so this is our limiting factor and so what I'm going to do right away actually is I'm going to do the following I'm going to separate out a separate service and I'm

[37:57] going to call this like the real time comment service um and what's going to going to be responsible for what it was before of this HTTP post request in request in order to create comments and then get requests to get old comments um

[38:13] but instead we're going to create the SS connections to this guy and so let me model that in a way that looks a little bit bit easier maybe can I make that transparent

[38:26] there you go so obviously this is going to be horizontally scaled too but what I'm trying to get across here is that there are a thousand of these servers right and I separated them so that I can scale them horizontally differently and

[38:39] so I might only need a hundred of these but a thousand of these and that's why I've separated them out right so every time an S connection gets established a user started watching a live video we are going to connect them to one of

[38:51] these realtime comment services and now you can see the issue hopefully a new comment just came came in our comment service got that comment and saved it to our database and now it needs to tell every single one of these 1,000 let's

[39:06] say to make this concrete let's say that new comment for video one just came in and so it needs to tell every one of these comment Services these real-time comment services that is connected over s to somebody who is watching video one

[39:22] that it got a new comment and so somehow we like need an arrow here uh it would somehow we need to tell one of these servers but how do we know which server

[39:34] that's the big question that comes up and so there's two ways to handle this let me show you the first here and the first would be that you would introduce a central dispatcher is maybe the way to think about this and so you'd have

[39:48] something for coordination zookeeper is a great technology to handle this coordination and so zookeeper is going to keep a mapping a registration of where all of these servers live and the live videos that they're responsible for

[40:03] is that when a new client wants to connect over ssse here it's going to service or realtime comment service either way and we're going to go over to zookeeper and say hey a new person needs to connect they're watching live video

[40:19] should I subscribe them to and it's going to give us a host IP address that we should end up connecting them to so that we can establish this SSE connection and so let me show you here

[40:31] what that might look like I'll put this over here maybe even I'll make it a different color right so that's what that's what exists here in Zookeeper it's this active mapping of all of the different servers and the videos that

[40:43] they're subscribed to and maybe how many users they have from each of those respectively and so now the process would be that when a new comment comes in the comment service let me go back to black the comment service is going to

[40:57] ask zookeeper hey I have a video for live video 123 uh which server should I message that to and it's going to tell then we're going to come down here we're going to draw an arrow to these guys

[41:10] have a new message or a new comment that came in and they can push it over to their friends over SS again via that inmemory mapping and so really quickly don't ask zookeeper every time that's slow uh we have a cache and so in memory

[41:26] on the comment service we've cached this mapping this Json blob so when a new comment service comes up it asks zookeeper the source of Truth hey what's me cat that on my server and then I'm going to subscribe to any updates and so

[41:40] any updates to zookeeper are going to get pushed now and my comment service is going to accept that incoming update and update its cash respectively but tying This Together comment comes in comment service says either by looking at its

[41:53] cash or asking zookeeper directly I had a new comment for live video video one which servers have ssse connections to everybody viewing live video One it's might be multiple I'm then going to make a request to that server with the

[42:08] comment and then that comment is going to look up in its inmemory mapping which connections it has for that live video and it's going to push them over ssse of course this should be unidirectional as well uh over ssse

[42:22] to the client so the client can render that new live comment so that's option one that's the the central dispat dispatcher approach there's some most obvious one is that we have an extra hop here every time that we come

[42:38] in we need to ask zookeeper and so we largely got rid of that hop here but we didn't on the connection when a new thing is connected it needs to ask zookeeper which should I connect to and so again the cach is helping us a lot

[42:51] here but there are some potential downsides there I'm going to hold my hands up and say this is hand waving honestly the cash is probably fine um but you have an extra component to manage uh potentially an extra hop here

[43:04] complexity in handling changes to the real-time comment services so these are going to go up and down respectively they might die for some reason we might need to scale horizontally and add more and zookeeper needs to maintain

[43:17] visibility of all of that state and so when new servers come up they have to be registered within zookeeper zookeeper needs to be pinging heartbeats to each of these to know who is still up and who isn't and so it's not just that these

[43:29] connected to or at least the number of people they're connected to but that needs to remain consistent with what zookeeper sees and so we've introduced a little bit of a consistency issue here that has the potential or capability to

[43:44] get out of sync and so zookeeper has a lot of great tooling to make this problem not much of a problem but these are the potential downsides here so that's that approach let me copy this and show you what the other of the two

[43:57] approaches would be so alternatively we don't have zookeeper instead of having a single source of truth that we go ask like an all- knowing Oracle hey where should I send this to instead what we

[44:09] can do is we can just broadcast so think about the comment service just like yelling out into the ether maybe with a megaphone it just says hey I just got a new comment for live video One and then all of these real-time comment Services

[44:22] have their ears open their ears to the street if you will and they're listening and if they hear oh live video One I have live video One then they'll take something with it and if they hear it and they're like oh I don't have live

[44:35] video One I don't care about that they won't do anything with it right and so in this case we would introduce Pub sub which is perfect for this sort of broadcasting and subscribing and so the comment service

[44:49] is going to get a new comment it's going to those arrows of the wrong way that's getting annoying um it's going going to put that new comment on pubsub probably after committing to the database so first for

[45:02] durability uh sake commit that new comment to the database then broadcast it by a pubsub so put it on pubsub this is just a channel right it's just like a message bus um that we're going to put this on and then all of these guys all

[45:18] three of them and you know when uh in production all up to a thousand of them or more are subscribed to this message bus right they're subscribed to this Pub sub they are subscribers so this is the publisher these are the subscribers and

[45:33] so a new video comes in it's going to yell via Pub sub new video came in and then all of these are listening and only those that hear one for which they know that they have an active user who is connected is going to take it and then

[45:47] again run that connection one. send with that new comment right so that's the the pubsub approach the nice thing here is that you don't have a consistency issue don't have two things that need to know their own State these real-time comment

[46:01] services are the only thing that need to know their state and it makes uh failures in the comment realtime comment service a lot more graceful you can imagine that one of these goes down well then those users just lost their s

[46:13] connection no big deal right and so they're going to need to reconnect different one of these we probably pulled another one up as well but different one of these the inmemory mapping is going to be updated and then

[46:26] because because we're still going over pubsub we're just still subscribed we're listening and we're going to send it over to them um so these guys know what they need to care about basically there's no other thing that needs to be

[46:38] in the picture so between these two options there is no right or wrong answer both of these options are great I tend to think that this Pub sub solution tend to think that this Pub sub solution is a little bit simpler um I think

[46:50] is a little bit simpler um I think actually that FB live comments in production went with they don't use keeper to my knowledge but went with a single dispatcher approach it's actually for slightly different reasons around

[47:02] scaling in collocation which we probably won't get into in this video but both of these approaches are absolutely valid in an interview if you're a mid-level get into this much detail if you're a mid-level candidate you'll say I'm going

[47:14] to use Pub sub to broadcast from the comment service to a real-time comment service which is connected over SS your websocket and you're golden if you're senior it would make sense for you to weigh these tradeoffs a little bit um

[47:26] should be able to articulate the pros and cons of each of these go into furthermore as we'll do here in just a moment talk about how each of these May scale respectively so now the problem that's worth discussing briefly and this

[47:40] staff or really great senior candidate talked about would be that all of our or are subscribed to every single new comment right now and so you look you

[47:52] can look at this as like drinking through a fire hose this is a pretty CPU intens task now because every single comment that comes in they're checking it's right for them if it is they're sending but there's a lot of wasted

[48:05] resources here because the overwhelming majority of the time it's not going to have an active ssse connection to for a client right and so we want to wait for them to just listen what to what matters for them and so what we can do is that

[48:20] we can partition our Pub sub and so if you're using like red is Pub sub these are called topics if you're using Kafka they're called channels but they're all of our Pub sub you can think of them as just like maybe oh kind of multiplying

[48:34] these out there's not a lot of space here but you get the picture uh kind of like a stack of different message buses here and so what you can end up doing is just separating using something like the hash of the video ID modul the number of

[48:48] channels that you want or the number of topics that you want so something like that and that's how you can determine which one of these it should go on and Services don't need to subscribe to every single topic or every single

[49:01] Channel they only need to subscribe to the sub set that they have an active client connected to them with right so you might only subscribe to a couple of these and this way you're only listening to messages that come in that are either

[49:15] your live video ID or something else that's in this hash grouping so it's not perfect it's not like it's only your videos uh someone might be tempted and I hear this in interviews they're like I'm going to partition based on

[49:27] um having a topic for each live video ID this just wouldn't work um it certainly is is limit off the top of my head but it probably wouldn't work as well limits the number of topics or channels that you have and so that's why you want

[49:42] to do this hash modulo end thing so you can just group them together and go from that end channels might be a thousand it might be a million uh whatever is needed always be tweaked but then we're only listening to the ones that matter now

[49:58] and I'm going to go over this just really briefly is that you don't want to have let's say that there's three people watching video one you don't want them they'll all show up on a different server here and now three different

[50:13] servers need to subscribe to the channel for video one when we could put all those on the same server we could collocate those ideally you have every single viewer of a single live video on a single real-time comment service now

[50:27] million concurrent viewers or something like the Chewbacca mom or a World Cup you're going to need to allocate that over 160 of these but at least it's 160 and not 500,000 right um so that's one optim

[50:43] case of Zookeeper approach this actually makes this easier uh because when a new ideas and we're just going to tell you to connect to that same box so this is

[50:55] great for that approach it's a little harder here the way that this works and it's a it's a bit complex so again not going to go into too much detail but you can actually have a zookeeper or

[51:07] zookeeper or another um another service similar that sits here and then your API Gateway via a layer 7 low balancer can have in the header what the live video ID is and then ask this guy where should I route

[51:20] and so it's similar to the dispatcher approach we're just only using the dispatcher uh to do Cod location you could make the argument at that point then like why not just come up here and do the dispatcher approach um like I

[51:33] do the dispatcher approach um like I said both work no Pros or no no right or weighing trade-offs the last thing I'll touch on um is that if you're a staff candidate maybe a fantastic senior candidate

[51:46] like for example what is the difference between using redis or Kafka um you actually probably don't want to use Kafka here because Kafka has a lot of from channels and you can imagine that when users are scrolling through their

[52:01] unsubscribing really quickly if they're not clicking into a live video they're just scrolling through their feed and so pubsub uh with reddis being in memory better approach there right that's the sort of thing that you can talk about to

[52:13] sort of thing that you can talk about to show staff level signal um I guess we mentioned that we would come back to that and I think now is an appropriate time as it pertains to scale so we said that we would have the the 1 million

[52:27] videos um let's just estimate and say there's about 100 comments per video I probably even high the overwhelming majority of videos probably get no majority of videos probably get no comments um this is concurrent so

[52:39] loosely let's say each video is about 30 minutes that's probably too long but in any case that would mean that if they were fully sequential this is like 48 um

[52:51] is I guess a day right so this is how many we're having a day um it's going to grow quickly there's a lot of Rights here and this is decent scale of Rights here and this is decent scale 480 million right is that math wrong 480

[53:06] million I think I'm off by one order of magnitude there so 4 billion um I don't spot in a video for everybody who's watching is difficult so forgive me but

[53:19] this is a lot it's also high right through put so we're going to want a right right through put but realistically a postgress database if sharded and scaled correctly can do this SQL databases would work fine but there

[53:34] are databases that are optimized for this sort of thing and given that we probably have a user and a live video ID column uh in our query is pretty simple it's just going to be able to query a primary or not a primary key uh but

[53:48] query an index here for for video ID and then you know we can use the created ad as our sort key so Cassandra is going to work well here um it's going to be really easy to Shard based on video ID so we can scale this horizontally like I

[54:04] said sort key on created at the query is simple it's just to get it for a given video ID and it's great at handling the high right through puts um so that's what i' probably do there and of course this is going to

[54:19] result in eventual consistency in places which is fine so this goes back to our to prioritize availability over eventual consistency so if we go and we try to hitting a read replica or something it might not have the latest comment that

[54:34] to show it to the user anyway without that latest comment no big deal and if anything we have the pub sub or the dispatcher um real time flow to make up for things there okay uh let's take a look back we

[54:48] our non-functional requirements we went through the full flow we nailed the interview we're hired we're excited hopefully this made sense the things about this interview just to recap is when to use websocket versus ssse

[55:01] necessary for bidirectional communication we have a one to one in the read to writes SS is unidirectional lighter weight but this means that we can only push from server to client um so it's great when you have a lot more

[55:14] uh Communication in that direction right and the balance isn't even there uh we talked about how to scale up whether it's pubsub or a central dispatcher in order to make sure that we can locate the correct real time server for which

[55:29] the persistent connection is on this is a common problem this is similar to what you're going to do in something like a real-time messenging app for example talk a little bit about scaling databases um the inmemory mapping I

[55:43] think we've done a good job here all right well done if you made it and you have a pretty good understanding of how to design a live comment feature for Facebook or any other live streaming platform if you like the video give us a

[55:56] like give us a subscribe if there were questions if I did something stupid let's chat about it I'd love to to respond to as many comments as I can um and like I said keep an eye out more videos coming to start 2025 but all in

[56:10] all hope you guys enjoyed and best of luck with your interviews talk soon