Message Queues in System Design Interviews with a Meta Staff Engineer

[00:01] new here, I'm Evan. I'm a former metastaf engineer and I'm the current If you're currently preparing for are if you landed here, then head over to Hello Interview. We have just about

[00:14] the overwhelming majority of which is totally free. Uh, in this video though, and specifically, as always, in the context of system design interviews. And motivating example that'll show you why QES exist in the first place. And then

[00:29] we'll talk all about what a message cube actually is, how it works under the finally, we'll get into those deep dives that interviewers love to probe into things like scaling, back pressure, ordering, durability, all of that good

[00:42] stuff. So, without further ado, let's get after it. Let's start things off with a little motivating example. I want you to imagine that you're building a the user uploads a photo to your server

[00:54] and then it needs to do a bunch of stuff with that photo. And so, for example, different resolutions. You may need to apply filters, maybe run some kind of content moderation checks in the

[01:06] appropriate, it's not nudity, those sorts of things. And each of those operations takes a couple of seconds each. In the simplest possible architecture that you could design, what might happen here is that the client

[01:18] uploads the photo to your server and then your server does all of that work synchronously. That single server handles all of that work. Resizing the the moderation checks, and then once all of that's done, and only once all of

[01:31] back to the client with the success message. This this works kind of uh but it has some real limitations. The the first is latency. The user hits upload and then they just stare at a spinner or whatever

[01:45] you put on the client while all of that processing is happening and they're just six seconds just to get that confirmation that they're they're they uploaded their photo, right? The second problem is around how fragile this is.

[01:58] What happens if the filter service maybe crashes halfway through all of that processing? Well, the whole upload now fails. Now the user ends up getting an error with a message maybe to retry and that resizing work that we already did

[02:12] is now totally lost too and we have to start from scratch. And then the third one is is a big one. How do you manage really bursty traffic? So, imagine that, for example, your app gets featured in the app store. Super cool. Um, but now

[02:25] your uploads spike from what was maybe 50 a second to 5,000 or 50,000 or 500,000 a second. And your servers, they can't handle more than maybe 200 per can't handle more than maybe 200 per second. And so those other 4,800,

[02:39] right, they're just timing out or they're failing or the user gets an error. your system is basically falling over under the load because it can't over under the load because it can't handle this new huge throughput.

[02:51] you how we can solve each of these problems by introducing a message cue. immediately, what if when that upload comes in, your server just saves the

[03:03] file and writes a message to a queue. The message is something simple like, The message is something simple like, "Hey, photo 456 needs processing." And back to the client saying that it's done. The upload's complete from the

[03:16] them maybe the single resolution photo just to them while the rest is the other end of the queue, you have a pool of workers, maybe a worker servers, right? Where each worker is pulling one of those messages at a time off of the

[03:31] queue and processing it. So if three photos get uploaded around the same different worker on the other end of that queue called consumers. And they'll all be able to process in parallel. So we can look at what happens in this case

[03:45] to our three problems. Now uploads are fast because your server is just saving queue. It's not doing the expensive processing. Failures are isolated because if a worker crashes while processing a single photo, that message

[03:58] and it picks up where the other one left off. And then on those big traffic spikes, it just means that the queue gets a little bit deeper, right? There's messages sit there potentially waiting to be processed. So at worst, there's a

[04:12] delay, but none are dropped or aired out. All right, now let's formalize what we just saw. What actually is a message Q? Well, it's pretty much exactly what it sounds like. A message Q is just a buffer, a queue that sits between a

[04:25] producer and a consumer. The producer is the thing or the server, the service that creates the work. In our example, this was that server handling the upload from the client. And the consumer is the thing that actually does the work. In

[04:38] our case, this was that pool of workers that process the photos. The way it works is really straightforward. The producer sends a message to the queue and then totally forgets about it. It doesn't care when or even if the message

[04:50] gets press processed at all. That's not its responsibility. It just fires and pulls messages off the queue and processes them at its own pace. So the Q's whole job is just to hold on to these messages until somebody is ready

[05:03] to deal with them. It's just a buffer in between the two services. The key producer and the consumer don't know about each other. They don't need to. And so this allows you to scale them independently and swap one or the other

[05:16] without affecting each other directly. You need more consumers, fine. You can producers where they were. you need more producers, you can scale those up while leaving, you know, the one or two consumers that you had. Um, to wrap up

[05:30] this this kind of intro on what is a message cue, uh, and I'll al albe it from school, which may help you, and I've seen this click for candidates, is waiter takes your order and then puts it on the ticket rail. The cook grabs the

[05:45] ticket off of the rail when they're ready, and the waiter doesn't just stand to make your food. instead they go and they serve other tables, right? And so the house from the back of the house exactly the same way a message Q does

[05:59] for our producers and our consumers, right? Pretty straightforward example. is, let's talk a little bit about how it actually works under the hood. Uh there's a few mechanisms that you need to understand and each one solves real

[06:13] acknowledgements. Let me give you a scenario. A worker pulls a message off the photo. Halfway through the worker crashes for whatever reason. It runs out of memory. Anything can happen here. And so what would happen to that message?

[06:28] message the moment the worker grabbed it, then that photo is now gone and it will never get processed. That user that user's upload is basically just lost forever. Uh which would clearly be be really bad. This is what

[06:41] acknowledgements solve. When a consumer pulls the message off of the queue, the queue doesn't delete it right away. Instead, the consumer has to explicitly send an acknowledgement or an act back to the queue saying, "Hey, I'm done with

[06:53] this one. You can go ahead and actually delete it now." So, if a consumer crashes before sending that act, the que assumes it wasn't processed and it ends ends up being redelivered to another consumer so that nothing was actually

[07:06] lost. Great. So, the the Q holds on to messages until consumers act them, right? acknowledge them. Um, but think about what this actually means, and you may have already thought this or recognized this. While worker A is busy

[07:19] processing a message and it hasn't acted yet because it's in its 4 seconds of doing all of that processing, then that message is technically still in the queue. So, what's stopping worker B from grabbing it, too? And then you'd have

[07:32] which is wasteful at best, and in some cases maybe even even dangerous or hurts the state of your system. Now, different queuing systems solve this exact problem different ways. In SQS, which is Amazon's native message queue, when a

[07:47] consumer picks up a message, it becomes invisible to all other consumers for a seconds. And so, if the consumer finishes and it acts within that window, then all good, the message was removed. If it doesn't, the message becomes

[08:01] visible again automatically and another consumer can retry. So, it's basically like a 30-se secondond window or a configurable time period. um where we we don't let consumers um double process a message. CFKA on the other hand, they

[08:14] take a pretty different approach though. They just assign each partition, we'll talk about what those are in a second, to exactly one consumer in a group, so that there's no even competition in the first place. There's only one consumer

[08:26] ever reading from any given logical queue, if you will. And then Rabbit MQ, talk about at the end, they use channel level prefetch limits and act timeouts all those details, but what's good for you to know is just that um all of these

[08:42] cues have a way to prevent duplicate processing. And whatever your queue of how exactly they do that. So, while the implementations differ, the concept is always the same. Every queuing system just needs a way to make sure that a

[08:55] message is only being actively processed by one consumer at a time. Okay. But even with axe and that duplicate uh message prevention, there's still a guys aware of. So what if the worker

[09:10] processes the message successfully, but then crashes right before that moment, it sends the acknowledgement. It sends the act. Now the queue thinks that it was never processed, so it redelivers it to another consumer. And that same photo

[09:22] gets processed twice. And maybe in our case, when you're just resizing photos doesn't matter at all. Doing that twice has has no bad effect. But what if the message was something like, "Charge Evan $50 for a banking application or

[09:36] something." Um, that duplicate message now just resulted in me being charged $100. And and that's obviously not cool. Um, this is the problem that's referred to as we talk about message cues as delivery guarantees. And there are three

[09:50] that you should know about and that your interviewer might ask about. The first and by far the most common delivery guarantee is called at least once delivery. This just means that the Q guarantees that every single message

[10:03] guarantees that every single message will be delivered at least one time, but it might be delivered more than once. The implication of this is that your consumers now need to be what's called item potent. Item potent just means that

[10:17] processing the same message twice will produce the exact same result. Let me give you an example. If your message says set user 123's profile photo to photo 5, then running that twice is fine. You get the exact same result. The

[10:31] photo five. The second time you run it, it just set it to photo five again, which it already was, so no big deal. But like we saw with that earlier message or with the banking example, if your message is something like increment

[10:43] user 123's post count by one, then running that twice is a problem. You've now obviously incremented their post count by two. And so in practice, you design your operations to be naturally item potent like in a photo example or

[10:57] you check whether the action has already be been completed before doing it again. And so if we if we take that example of increment user 123's post count by one, the message or the work would actually be uh update user 123's post count to 54

[11:13] where 54 was what it was before plus one. Right? That way running it twice the outcome is always still 54. Um, now this at least once delivery with item potent consumers. This is almost always the right answer both in production and

[11:28] use, it's almost always the right answer to say I'm going to have at least once uh delivery guarantees and I'm going to make sure that my consumers are item and potent. The second delivery guarantee is what's called at most once. And this is

[11:41] consumer takes a message off of the queue, we immediately delete it off the queue at that moment. if something goes wrong, you know, at most one guy processed it. Uh, at worst, nobody processed it. And so, you really only

[11:55] analytics events or metrics where losing a few data points is totally acceptable. You have to be able to accept the loss if you use at most once. And then the third one is the holy grail. It's exactly once. Uh, this is that every

[12:08] message is processed exactly one time as it says, right? Uh the reality is that true exactly once delivery is extremely hard to achieve in distributed systems. CFKA supports a form of it for specific patterns within its own ecosystem. But

[12:24] it comes with real trade-offs and limitations. And so my honest advice to you is don't promise exactly once in your interview unless you can explain once with those item potent consumers is the safer and frankly the more practical

[12:38] production. Now that we understand a little bit about how Q's work and what question is when should you actually reach for one? Well, there's four signals that you should end up looking for that may lead you to introduce a

[12:52] very similar to the motivating example that we started with. The first one is that async work. And so this is when a user doesn't need an immediate result like sending an email, generating a report, processing uploads, all those

[13:05] things that we talked about. Um the litmus test is really simple. Ask yourself, does the user need the result of this operation right now or can they wait a little bit? If the answer is no, then it's a great candidate to put it in

[13:18] a queue and have a worker process it asynchronously. The second is the bursty traffic. Like we saw in our example, you need to absorb spikes in traffic without dropping any requests. And so the queue is there to kind of smooth out the load

[13:30] by acting as a buffer and just accumulating a backlog of work that the consumers can get to, you know, when they have time or they have resources. Third is decoupling. This is your producer and consumer might have

[13:42] completely different scaling or hardware needs. Going back to our image are super lightweight. They just accept the file, drop it into a message queue, but the workers doing the actual processing might need GPUs or beefy

[13:56] with a queue between them, you can scale and provision each side independently. You're not forced to run expensive GPU instances just to handle uploads. This can make it more costefficient.

[14:09] And then the fourth one is reliability. When you just can't afford to lose work. temporarily unavailable, the queue holds online. And you make sure that you never lost anything.

[14:24] just really quickly because I see it in quite a few interviews, especially with it's be careful introducing a queue into a synchronous workload. It just doesn't latency requirements in your non-functional requirements, like sub

[14:38] 500 millisecond response times to get an answer, by adding a queue, you've nearly that constraint. Not only do you have a bunch of complexity on figuring out how to get that message back to the clients in the first place, but you've broken

[14:51] by the nature of the way that these systems work. So again, cues are for that work that you can afford to do later, even if later is a few seconds Okay, now let's get into some of those deep dives. This is everybody's favorite

[15:04] part, right? The things that the interviewers really love to dig into system. So, it's really important that you're prepared for these questions, the whiteboard. Here's what the interviewer might come at you with.

[15:17] Okay, so the first one's about scaling. They might say, "How does your queue handle the increased throughput or handle increasing throughput?" And what single queue can only handle so much. When you need more, you do what's called

[15:31] partition. And partitioning just means splitting the queue into multiple independent sequences or like kind of sub cues of messages. Different workers can then process different partitions in parallel so that your throughput scales

[15:45] horizontally with the number of partitions. So by splitting into quotes just because we're kind of uh conflating terminology here, but logical cues, you can have multiple consumers consuming from the same time

[16:00] and thus increased the throughput. Um on the consumer side, you have what are called consumer groups. And a consumer group is just a pool of workers that themselves. So if you have six partitions and three consumers in a

[16:15] group then each consumer can handle two partitions. You need to go faster. Well then you can add more consumers. Now the um importantly there is a ceiling here. have partitions. If you have six partitions and six consumers adding a

[16:31] there's no partition left for that new consumer to consume from. And hopefully Now, really importantly, the partition key, which is how you decide which

[16:43] message goes into which of these partitions, is really, really important. It's analogous to choosing a shard key or partition key in a database. And it matters for two main reasons. The first is ordering. Messages with the same

[16:55] partition key always go to the same partition. And within a partition, ordering is guaranteed because they're sitting in that same subq. So imagine again that you're you're processing that bank transaction example we brought up.

[17:09] A user maybe deposits $100 and then withdraws $50. Clearly they should be able to do this. They put $100 into the bank. Now they want to take 50 out. If those two messages though ended up on different partitions,

[17:22] then the withdrawal could get processed first and now it's rejected because at that moment maybe the account is empty, right? And so by using account ID as the partition key now both that both of those messages will land on the same

[17:36] same account. They'll be in the right order and they'll get processed in that right order where we add the 100 before we take away the 50. Right? And then the second one that's important about choosing your partition key is the even

[17:50] distribution. You want your partition keys to spread work ac across these partitions pretty evenly. You can imagine that if you're building like a ride sharing application and your partition by city then now New York City

[18:03] is going to be excuse me New York City is going to be absolutely slammed while Boise sits there maybe getting nothing right and so that's what we call of course a hot partition and you'd probably want to partition by something

[18:16] more evenly distributed like a right ID so that you don't have that hot doing all the work where another consumer is just sitting there waiting because it's just watching Boise and there's nearly as many Uber rats, right?

[18:28] here, though, and it's one that you might discuss in your interview, and it's that the key that gives you ordering might not always be the key And so, choosing the right partition key here is one of those decisions that's

[18:42] interview thinking through the consequences on those two factors, ordering and distribution. The the next deep dive that interviewers love to ask is what happens if your producers outpace your consumers. So if

[18:56] faster than your consumers can process them, the crew the the the queue just grows and grows and grows and grows and grows, right? And the queue doesn't solve a capacity problem. It just delays it. It's just buying you time. If you

[19:11] receive 300 messages per second, but your consumers can only handle 200, that queue is growing at a 100 messages every single second and you'll never be able of memory on that queue and things are going to go wrong. So, what do you do or

[19:26] things. The first of course is scaling. You can you can add autoscaling and a lot of the uh cloud providers provide autoscaling where you monitor the queue depth and when it starts to grow too much you spin up more consumers so that

[19:39] you can consume quicker or you can even add additional partitions to the queue. interviewers are oftent times looking for is to apply what's called back pressure to the producers themselves. And so slow the producers down.

[19:54] messages or maybe returning an error to the client saying, "Hey, we're a little overloaded right now. Please try again in a minute." And then the third and should just be setting alerts of course on your QEP so that you know when

[20:07] good to have the monitoring and the alerting here. Um be ready be ready to discuss this one. Interviewers want to know that you understand that a Q isn't just magic, right? It's a buffer, not a solution to

[20:20] insufficient capacity. Moving on. Uh, sometimes a message just interviewer might ask you, what happens when a message fails to process? Maybe the image file is corrupted or the downstream service is temporarily

[20:35] unavailable. That's fine. It happens, right? So, think about our photo upload example. What if I just uploaded a corrupted photo? It will never succeed. No matter how many times you try, it will continue to fail. And this is a

[20:47] really common problem and it's what's called with message cues a poisoned is, but it's just a malformed or problematic message that crashes the consumer every single time and there's no recovering from it. So without guard

[21:00] rails, it's going to retry forever. And meanwhile, everyone else is just stuck behind it waiting. It's just consuming all the resources of the given consumer To solve this, most queuing systems, they let you configure a max retry

[21:14] count. The message gets tried, tried, tried again, maybe five, six times. And if it still hasn't succeeded after five tries, instead of retrying forever, you shunt it and you put it to a dead letter Q or a DLQ. And a DLQ is just a separate

[21:30] Q or sometimes even just a separate partition where failed messages go so figure out what went wrong. Meanwhile, the main queue keeps moving. So if this main queue. put it somewhere else where we can just wait and maybe an admin

[21:44] there and try to fix those, right? Or nowadays an AI model. Um, mentioning great. It shows that you have a bunch of seniority. It shows that, hey, I'm going add a dead letter Q here. Um, this shows

[21:58] failure scenarios. Now, speaking about failure scenarios, scenario? What happens if the Q goes down? Um, and this is a question about non-functional requirements. It's

[22:12] talk about. And it's good to know that modern message cues, at least some of them like CFKA, they persist messages to disk and they can replicate them across in the CFKA ecosystem, but these are just different servers. So if one broker

[22:27] just like with read replicas with databases. Same concept. So in this way, no messages are lost. Uh, CFKA in particular is interesting here because it stores messages on disks with a configurable retention window. So you

[22:41] could keep messages around for say a day, a week, even forever if you wanted to and you had the capacity. And what this means is that you actually can replay messages from the past too, which is a really powerful recovery scenario.

[22:54] Uh, if you need to reprocess data after kind of something went down, you have a you can imagine that your consumers go down for a while, they're offline for an just backing up. No big deal. Because when your consumer comes back on, it can

[23:08] start to just take all of those things again. Even more so, if that consumer was broken and it processed things incorrectly, well, we can just put a new consumer in, tell that consumer, hey, reprocess back from an hour ago, even

[23:20] though the messages were consumed. Um, and we can kind of fix what we broke. Speaking of CFKA, let's maybe quickly go through what are the most common message Q technologies. You certainly don't need to know all of these for your interview.

[23:34] that you're comfortable talking about. If you don't have a default already, choose CFKA. It's kind of like the interviewing industry standard. Um, but them. Of course, starting with CFKA. CFKA is probably the most widely used

[23:49] recommend. Um, it's a distributed streaming platform that can act as both a message cue and a stream processing system. So, it's built for really high we talked about. It writes messages to

[24:02] partitions. Right? When we're talking about partitions, that's how CFKA works. It also supports those consumer groups. But one thing that makes CFKA a little different from a traditional queue is that messages aren't removed directly

[24:15] after they're consumed. So this was that example where you want to replay messages in case something some code in the consumer changed, right? In CFKA, they stick around for those retention periods. So multiple consumer groups can

[24:27] read from the same data independently and you can replay messages if you need to. Like I said, um the second one I'm going to call out, breakdowns quite a few times, is SQS. This is Amazon simple Q service. It's

[24:41] the AWS hosted ecosystem version of a message Q. Um it's simple, it's fully need to worry about. It comes basically in two flavors. A standard Q which gives you the best effort ordering and really high throughput or FIFO cues which give

[24:56] you strict ordering but at a lower throughput. So SQS is a great choice straightforward, you don't need more advanced features and your interviewer is cool with you pulling uh hosted solutions from cloud providers.

[25:09] The last one that that I'll mention is Rabbit MQ. And RabbitMQ is a more traditional message broker and it supports complex routing patterns bindings. Obviously, I'm not going to have time to get into that here, but

[25:21] it's all the same underlying con uh concepts. It's maybe less common in be honest with you. I hear it less, but it might be worth knowing that it exists uh especially in cases where you need some sophisticated message routing

[25:34] section, if you don't already have a go-to, pick CFKA. It's the most versatile. If you want just hyper simplicity in the AWS ecosystems around, timeout thing that we had mentioned earlier. Uh it has the option for

[25:48] ordering guarantees or looser guarantees but higher latency. All right, there you have it folks. Um hopefully you found this useful. Just a really quick recap. Message cues decouple your producers from your

[26:00] consumer. They buffer bursty traffic so nothing gets dropped. They distribute work across pools of workers. Um, you should know when to use them, know the pitfalls, be specific and detailed when you bring things up. Um, any questions,

[26:14] ahead and please drop those in the comments. I respond to as many of those drawings that I use in this video down in the description as well. So, go ahead LinkedIn. Connect with me, send me a message. Again, I respond to as many of

[26:29] those as I can get to as well. I love hearing from you all, especially if you interviews. I'd love to celebrate with you. Uh most importantly, good luck with going to do great. You're going to nail it. You're putting in the work. Uh and

[26:43] it. You're putting in the work. Uh and I'll see you soon.