[00:01] I'm Evan. I'm a former Meta staff engineer and the current co-founder of hellointerview.com. If you're preparing for software interviews, head over to need, overwhelming majority of which is free.
[00:13] we're going to be covering the basics of caching, and specifically in the context we'll look at where caching fits into a system. We'll talk about the most common caching architectures, typical eviction policies you want to understand. We're
[00:27] issues that show up when you introduce caching. These are things like consistency, stampedes, hot keys, the things that interviewers really love to prepared for. And then finally, we'll go over how to talk about caching in an
[00:40] depth you should go you should go into. And the things that interviewers are are usually looking for. So, this should be fun. Without further ado, let's get after it. Chances are you already have a pretty decent idea of what caching is.
[00:52] just the very basics really quickly to make sure that everybody's on the same And so, a cache is quite simply just a temporary storage that keeps recently used data handy and close by so that you can fetch it faster the next time.
[01:07] So, to see why this matters, let's take a look at an example here. And consider the difference in speed between where data usually lives in a database and where it can live in a cache. And so, accessing data from disk, like
[01:19] an SSD in the case of a database, takes about a millisecond on average. Accessing data from memory or RAM on the other hand takes about 100 nanoseconds. This is roughly 10,000 times faster. Now, that gap adds up really quickly
[01:34] when you're serving thousands of requests per second. And caching takes advantage of that big difference. It keeps copies of frequently used data in a faster layer, often times memory, but not always. We'll talk about that later
[01:46] on. So, that systems don't have to reach all the way back into that slower source So, you have the basic idea. Caching trades a bit of storage and complexity Now, the next question is, where should you cache your data? And there's a few
[02:00] caching can live, each of which have their own set of trade-offs, of course. most common in system design interviews, This is where you introduce a dedicated caching service like Redis or Memcached.
[02:15] It runs, importantly, on its own server and manages its own memory, and it's or your database, right? It's your own component in the system here. And so, when your application needs data, it first checks the cache. If the
[02:28] data is found there, that's a cache hit, and it returns your data instantly, super fast. If it's not there, we call that a cache miss, and it has to fall back to the database, fetch the data, and it stores a copy of that data back
[02:40] in the cache, and also returns it back to the client, right? Uh now, the nice thing here is that in a scaled system, which might have multiple application servers like we represented here, all of these different application
[02:52] cache. This way, once one server has fetched and cached the data, the others can all reuse it instantly instead of all hitting the database separately, right? Because this is a global view. It's a
[03:05] global cache that's shared by all of the different application servers.
[03:20] is what we call in-process caching. And this lets you skip the complexity of adding something like Redis entirely. Uh it's often overlooked, to be honest, and the context of interviews, it's probably overlooked more than it should be, but
[03:35] Now, the key thing to note here is that on really big machines nowadays that have plenty of memory. And you can order to cache data right inside of the process.
[03:48] process. Uh this is important because it's by far don't have to go and have an expensive network hop here to hit some external cache anymore. The data is already
[04:01] sitting in the same memory space as your application, right? So, you don't have right there where you want it. But of course, this this comes with trade-offs. the external cache, each application server has its own in-process memory.
[04:17] caches something, the others won't see it. So, you can end up with these inconsistencies or even wasted memory if you're not too careful. context of a system design interview, that you probably won't need to bring
[04:31] this up unless you're talking about a low-level optimization or have a use case where ultra low latency matters. For example, if you need to cache config data or small lookup tables that every single request depends upon, then
[04:44] caching within the application server makes sense, but your default should remain as external caching. Next up, we have what are called CDNs or content delivery networks. And so, a CDN is a geographically
[04:57] distributed network of servers that can cache content closer to your users. And than it is. It's just putting servers around the world so that they're close not optimizing for the difference between memory and disk speeds like we
[05:12] were before. Instead, we're optimizing for network latency. has to travel all the way to your origin server. If you had a server in Virginia, like was the case here, think that this is S3
[05:25] user is all the way over here in Australia, then this round trip could take 300 to 350 milliseconds. That's huge, especially when we're talking millisecond. With a CDN on the other hand, that same
[05:40] request might hit an edge server that's just a few miles away, which may be 20 to 40 milliseconds round trip, which is a huge difference. a user requests something like an image, that request goes to the nearest CDN
[05:52] that image is already cached there, it's returned immediately. Perfect, that's the happy case, that's the cache hit. If not, if it's a cache miss, then the CDN itself goes and fetches that media or whatever you're looking for from uh
[06:07] blob storage you have, and then it's going to return that back to the CDN. The CDN will then cache it so that it has it for next time and return it back to the client. Now, modern CDNs, and this is something
[06:20] lot more than just cache static media, which is what they're most known for. They can also cache public API responses, of course HTML pages, run edge logic even to personalize content. But as far as a system design interview
[06:34] goes, the most common and the most impactful use case to bring up is on media delivery. So, things like images, videos, or static assets, files, etc. around the world. And so, if you have global users who are accessing media
[06:47] regularly, then a CDN is probably a great fit for you. client-side caching. So, this is when data is stored directly on the user's device, either in the
[06:59] browser or the app, which avoids unnecessary network costs. Um So, in web apps, that might be something like the HTTP cache or local storage within the browser itself. For mobile apps, this could be data kept in memory
[07:13] device. And it's nice cuz it's obviously super it comes with the downside, of course, and that's that you have less control over it. Data can go stale, validation, freshness, all of that is a bit harder.
[07:27] And so, when it comes to your interview, you'll see this come up a lot less often. Um usually, it's only relevant when your system involves some offline functionality or client-heavy workloads. For example, if a browser's reusing
[07:39] images it already downloaded, or an app like Strava caching your run data locally while you're offline and then syncing it once you're reconnected. We have a problem breakdown where we do exactly that. But for all intents and
[07:52] for you to know as it pertains to your system design interviews. popular feature on hellointerview.com, guided practice.
[08:05] practice system design interview step-by-step using that hello interview through everything from the non-functional requirements to the core entities, API routes, all the way through to your high-level design and
[08:17] deep dives, drawing on the whiteboard and narrating your response. All while you're doing well and where you can improve by a model that Stefan and I have spent hundreds of hours tuning. We expanded the library to 25 of the
[08:32] questions now, and we're constantly adding more. So, candidates absolutely love this feature. I think you will, too. Check it out at hellointerview.com. that we've covered where you can cache your data, let's talk a little about
[08:45] cache architectures. Cache architecture is just defined how the cache. And so, specifically, this is defining the order in which reads and writes happen between both your cache and your database and your application
[08:58] And so, the most common caching pattern by far, and this is the one that you should default to in your interview, uh is cache-aside. that the application checks the cache first. This is the one we brought up
[09:13] earlier, right? If the data is there, it returns it, that's a cache hit, great. If not, it goes and fetches it from the database, and then it stores it in the cache, and returns it to the user. And so, cache-aside is great because it
[09:25] keeps the cache really lean. You only cache data when you actually need it, that users actually requested. If a user never requested anything, it never made it into the cache. But the downside is that a cache miss is
[09:39] going to add that latency since the request then has to go hit the database, operation, store it in the cache, and return it back to the user, right? if you're only going to remember one caching architecture from this video,
[09:53] you're probably going to use in the interview, uh but it's important for us well. Starting with write-through caching. actually writes directly to the cache
[10:06] first. And then the cache synchronously writes that data to the database before write isn't considered complete until both the cache and the database have In practice, this means that you need a caching library or a framework for this
[10:20] part right here that supports this write-through behavior. Something that knows to trigger your database write logic automatically. Because tools like Redis or Memcached, they don't natively support this. And
[10:32] yourself in your application code by writing to both at the same time, and come from that, or you'll want to use a library, something like Spring Cache or Hazelcast, which can automatically do that write-through for you.
[10:47] Um but the trade-off here becomes pretty obvious, and that's that you have slower writes cuz you need to wait for both of these to happen. Um and then also, you can pollute your cache with all this data that may never
[10:59] be read again. If we write everything to our cache, then it might be data that nobody ever actually accesses, and we're just bloating our cache for no reason, a moment ago, right? Um and then write-through also suffers
[11:13] problem. This would be that if the cache update succeeds, but the database write fails or vice versa, then the two enter an inconsistent state. And so, you would need fancy retry logic, error handling,
[11:26] But in a distributed system, this perfect consistency is incredibly hard complications here. Now, interview? Well, the reality is that write-through is much less common than
[11:40] this specialized infrastructure and has all these tricky edge cases around And so, you'd really only bring it up when reads must always return the fresh data and your system can tolerate some slightly slower writes. If that's the
[11:54] case, this could be a good fit. But if your mind is going in this direction, really be sure that you can convince yourself that cache-aside or some other design system doesn't already satisfy your use case. Next is write-behind
[12:06] sometimes. And it's really similar to write-through, which we just discussed, synchronously, the cache writes to the database asynchronously in the And so, the application only writes to the cache, just like we did for
[12:20] flushes those updates to the database usually in batches later on. than write-through. We solved that problem, but it introduces new risk. And
[12:32] that's that if the cache were to crash or fail before this flush, then we would You would use this only when high write throughput is more important than immediate consistency. And so, for example, something like analytics or
[12:46] metric pipelines where some occasional data loss might be acceptable. pattern is useful to know, and if you are an expert in caching and you have a use case for it, by all means use it. If you're a novice,
[12:59] can strongly justify it. There are other ways to solve the problems that we just um and you're probably introducing or opening an opportunity for more would like. So, my honest suggestion to you is probably
[13:13] avoid it. The last one we'll discuss is read-through caching. It's incredibly one we talked about, except that the cache handles the database look-up instead of the application. And so, on a cache miss, whereas before
[13:25] from the database and updated the cache and returned the value, now the cache itself is going to do that. So, we try to read from the cache. If we miss, the the database, stores it in the cache, and returns it back to the application
[13:39] You can basically think about it like cache-aside, but with the cache acting And this is essentially how CDNs work, right? As we discussed a moment ago. When it when you have a CDN miss, it fetches from the origin server, caches
[13:51] time. Um now, as far as system design bring this up in the context of CDNs or edge caching. For most application-level default because it doesn't require a special framework or caching library to
[14:06] you can just use Redis, Memcached, something simple, um and not have to have some adapter right there. So, zooming out now, let's see if we can fit all four of them in there. There we go. We got cache-aside, write-through,
[14:20] you're anything like me, you're probably thinking, "These names are super confusing. How am I supposed to remember all of them?" And I've good news for don't care if you remember these exact names or exact terms. What matters is
[14:34] clearly. And so, if you forget cache-aside, no problem. Just say, "I'll there, I'll go to the database and then update the cache." That's really all we And so, it's always better to show that you understand how caching works than to
[14:49] terms. So, don't overwhelm yourself. Understand how cache-aside works most importantly, be able to describe it, don't stress yourself on the naming. cache eviction policies.
[15:02] memory instead of on disk is that memory is limited. You can't fit your whole usually can't. And so, you need a strategy for deciding what to keep and what to remove as new data comes in. And that's exactly what your eviction
[15:16] stay in the cache and which ones get replaced when it fills up. really straightforward, and so we're not going to make them any more complicated that you should know about, three of which you might actually use in an
[15:30] interview, one is just good to know because of its simplicity. And so, first, we have least recently used or LRU. And just like the name suggests, this eviction policy evicts items that
[15:43] haven't been used very recently. In practice, it's often implemented with a linked list or maybe a priority queue that tracks the access order, but you're level of detail that level of detail in a system design interview.
[15:55] Implementation for eviction policies is almost always out of scope, right? And so, the second one that you have here is least frequently used, LFU. Really similar idea, but instead of evicting based on recency, it's based on
[16:07] how often something is accessed. And so, the least frequently used items are evicted first, even if they were used recently, right? So, even if it was was a second ago,
[16:21] everything else that that might have been accessed more frequently. Um third is first in, first out. This is that simple one, right? It's exactly item gets removed to make space for your newest item. It's dead simple, and it's
[16:36] rarely the right choice in in a system design interview. Um and then lastly, you have time to live And so, here each cached item has an expiration time, and once that time
[16:49] passes, say like 5 minutes, then the cache will automatically remove it. This is great for data that can go stale, like user sessions or API responses or And so, in system design interviews, least recently used is the most common
[17:04] and oftentimes the default. Um least frequently used makes sense when your access pattern is highly skewed, meaning there's a few items that are read way more often than others. And then TTL is super common as it's
[17:16] perfect for when freshness really matters more than recency or frequency. straightforward. We store data in a faster layer, we read it when we can, and everything just gets quicker. But in practice, adding cache introduces
[17:29] they're challenges that interviewers love to ask about. And so, there's this some of you might have heard, and it's that there's only two hard problems in computer science, naming things and cache invalidation. And that second one
[17:43] caching, new problems start to show up, things like uneven loads, stale data, or unexpected spikes in traffic. And so, we're going to go through a few of the ready to talk about in an interview, the ones that interviewers will kind of be
[17:56] ones that interviewers will kind of be the most likely to maybe probe into. And so, let's start with what's called a cache stampede or oftentimes referred to And so, this happens when a popular cache entry expires via that TTL that we
[18:10] discussed a moment ago. And suddenly, a flood of requests all try to rebuild that cache at the same time. And so, even if that window lasts just a second, every single one of those cache misses is going to hit our database,
[18:23] turning one query into thousands or even millions, and ultimately overwhelming concrete example. Imagine that you have some website for which you cache the homepage feed with a TTL of 60 seconds, right? You don't want
[18:37] it to get too stale. Um so, 60 seconds is what you choose. And then we get 100,000 requests every second. And so, all 100,000 requests cuz all users need that home feed, they hit us
[18:49] in the cache, and everything works great. But after 60 seconds, it expires. And what happens when it expires in the case of cache-aside is that we're then and update the cache. But in that moment, 100,000 requests
[19:03] then they all go try to hit the database, and they could take the database down, overwhelm it, and cause cascading failures. Now, there's two common ways that you can prevent this. The first is what's called request
[19:15] coalescing or single flight. Both fairly fancy names, but the idea is actually really simple. When a request tries to rebuild the same cache key, or the same cache key, then only the first one should work, and the rest of them
[19:29] should just wait for the results to come in and then read from the cache. handle it. The second, which is actually maybe equally as common, I'll contradict myself, is called cache warming.
[19:41] instead of waiting for popular keys to expire, waiting that full 60 seconds, you can proactively refresh them just before they do. And so, say at the 55-second mark, we could come in here and refresh the feed, um thus giving it
[19:56] another 60 seconds, and essentially preventing it from ever actually expiring, right? We just keep refreshing it every 55 seconds so that it remains fresh and not stale, um but it never expires and causes this thundering herd.
[20:09] Next up is cache consistency. And this is probably actually the most common issues that interviewers like to ask about when caching comes up. And this database could return different values for the same data.
[20:21] right? Because most systems read from the cache, but they write to the database. And so, this creates this short window, depending on your eviction policy, where you can have stale data in the cache. All right, let me give you a
[20:33] Imagine that you have some social network, and a user updates their profile picture. And so, that new value is written to the database, but that old cache. And so, now other users who are
[20:46] requesting are hitting the cache, and they're getting image two. No, that's bad copy and paste, actually. Let me update that. Bang, right? All Image three. Wait, no, I want one image one. Silly. Okay.
[20:59] original one, image one, despite the fact that we'd already updated the database to image two. And they're going to continue to read this value until this ends up being evicted for some reason. Right?
[21:11] Now, there's no perfect fix to this. It very much depends on how fresh your data needs to be, and that is a case-by-case basis, right? But there's some common question if your if your interviewer asks about this, or if this ends up
[21:25] And so, the first is to invalidate on write. And so, if consistency is really important here, then when that profile picture came in and we updated in the database, then we could also go delete
[21:37] that key proactively from the cache. In this way, the next time a read comes missing, and we would go grab it from the database and update the cache, invalidating on write, and it's going to make sure that you're reading the latest
[21:51] data for the most part. You can also use just short TTLs. And so, if some stale-ness is acceptable, well, you can keep the cache entry here, is like what we had a second ago, right? So, in the case of that newsfeed, we had
[22:05] a 60-second TTL, and maybe we keep this around for 60 seconds, then it'll be we know this is something that changes often. That's potentially acceptable. you might just accept that eventual consistency is fine, and this is totally
[22:20] valid. Uh this is for things like feeds, analytics, metrics, or brief delays totally acceptable. And so, if I was discussing this specific problem that we have here, maybe I would say, you know, we have a 5-minute TTL on our profile
[22:34] data cached here, and that's totally fine. Because what it means is that some 5 minutes, and I'm okay with that, and it's fine if they see an old image for a while. Life is still going to go
[22:48] that I could justify based on the design, and it would be a totally valid justification to make. Another common issue, which you've seen come up in many of the videos if you've watched any of our other content, is called hot keys.
[23:01] gets way more traffic than everything else. Even if your overall cache hit planned, that single key can still become a large bottleneck for your And so, for example, imagine that you're building Twitter X, and everyone is
[23:15] viewing Taylor Swift's profile. That cache key for her user data could be receiving millions of requests per second, and that one key can overload a single Redis node or shard, even though that shard, or even though that cache is
[23:29] technically working as expected, right? And so, this, of course, isn't just a talk about this if you've watched our other videos in other contexts, too. partitions as well. But it's a common follow-up question in interviews once
[23:41] you introduce caching, especially if you introduced caching in order to scale your reads, right? So, caching does increase your overall read throughput. You're now hitting memory as opposed to disk, but it doesn't completely solve
[23:54] the problem if there's one piece of data that is so overwhelmingly popular, like And so, there's a couple things that you can do. The first and and the most common solution is to just replicate these hot keys. And so, if everybody is
[24:07] trying to know about Taylor Swift, well, you can put Taylor Swift on each of the different instances of the cache in your cache cluster as you've scaled up. Uh if
[24:19] clustering is not familiar to you, I'm going to link a video uh on sharding below, but this is basically the concept of a single cache isn't enough, so we needed to add additional instances of caches and shard
[24:32] our data across them, meaning split our data across them. Okay. So, now back to the issue at hand. Replicating hot keys, right? Means that we'll take Taylor different caches. And then now the application server can
[24:46] them. So, instead of all the traffic going to this cache or this node, it could hit any of the three here, right? Another fairly popular thing to do is that you can add a local fallback cache. Basically, use that in-process caching
[25:00] idea that we talked about earlier in order to add a cache Oh. here. Right? And so, this way, we keep extremely hot values like Taylor Swift in the app's
[25:14] memory, so that repeated requests don't ever even need to go hit Redis. And we just store them locally here. Um hot keys, let's see, to wrap up here, they were really good reminder that caching helps us scale reads, uh but it
[25:28] infinite, and interviewers love to probe you on that. So, be ready. probably the most interesting to all of you who are preparing for your system you discuss caching in your actual system design interview. Well, there's
[25:43] when should you bring it up? And then B, once you bring it up, how you should introduce it. Right? And so, the first and most important sake of adding a cache. I see this all the time. I see candidates who just
[25:55] throw it down without any proper justification, and sometimes they're wrong, and even if they're right, the lack of justification was a red flag. typically when one of four things is true. The first is that you have a
[26:09] straining your database. And so, you could say something like, "We're serving 100 million daily active users, each of which are making uh 20 requests per day. database." Somebody check my math. I I did that a little too quick in my head.
[26:24] our database can handle, so let's put a cache in front of it to take the read load off of our database. Cool. For expensive queries, you could say something like in the case of newsfeed, right? Computing a user's personalized
[26:36] joining a bunch of posts, followers, likes, all of these things across multiple tables, uh and that's going to be really expensive to compute. And so, what we can do is just cache that newsfeed, uh store it with a TTL of 60
[26:48] seconds or so, and then serve it really quickly from something like Redis. going to come up in an interview because certainly in real life, if your database is starting to peg out on its CPU, then
[27:01] front of it. And then latency requirements, this certainly will come specified that you needed a 100-millisecond response time on some API endpoint. Well, then you could argue or justify that that database query is
[27:16] has expensive queries, and so, you'll have to cache it instead, right? Identify the bottleneck, be able to quantify it with some rough numbers, and then explain how caching solves it. Now, once you've done that, you'll want
[27:32] to actually introduce caching, and this is where you talk about all the things that we discussed in this video. And so, the first one being, identify the mentioned a moment ago. And then decide what to cache. Not everything should be
[27:45] that is causing the issue, that needs to be read frequently, maybe it doesn't change often, it's expensive to fetch or compute. These are the things that you want to focus on caching. Think about what the cache keys are going to be. Be
[27:58] explicit about this. Especially in junior or more mid-level interviews, I add a cache here, and that's going to solve everything." And my follow-up is always, "Well, what are you caching? What is your cache key? Uh what values
[28:10] you're proactive about that. The second thing is to choose that cache everything that we were talking about with cache-asides, write-throughs, write-behinds, etc. So, you can say, "I'll use cache-aside on read. We check
[28:24] it. If not, we'll query the database, store the results in Redis, and then return it back to the user." Right? And then you'll want to mention the "We'll introduce a cache. We'll either use LRU, uh LFU, or maybe you'll mention
[28:38] that you'll have a TTL for preventing stale data." But provide some justification there as it is relevant to your system. And then lastly, address any potential downsides. And so, don't necessarily
[28:50] mentioned, but think about your system. Which of these are relevant? Do I have a TTL on a popular key that could cause a cache stampede or a thundering herd? Do I have issues with cache consistency, where I'm going to potentially be
[29:03] serving data that is no longer fresh, but instead stale? Uh and is that a problem for my system, or is it not? And do I have any issues maybe with hot keys If you discuss in this order these five things, then you'll be a caching pro,
[29:18] your interviewer will be impressed, um and you can continue to talk about I think the last thing that I'll mention is that caching usually comes up during scale. And so, when you get to your
[29:31] either talking about scale or latency, that's often times the most appropriate figure out where it fits into your system. And it's super common in these to have down pat. All right, there you have it, folks.
[29:44] Hopefully, If found this useful. Um any questions, anything you think I got wrong, go ahead and drop a comment. I respond to as many of those as I can. You'll have the Excalidraw that we used here in the description. So, go ahead
[29:56] and check that out. My LinkedIn is there. Connect with me. possible. I'd love to hear success stories, especially if you're finding what we're doing useful. And most importantly, good luck with the upcoming
[30:08] it. You're putting the work in. See you soon.