4 Ways to Fix Cache Stampede
42sQuickly outlines a common problem and four solutions, making it highly educational and shareable for developers.
▶ Play ClipAI Summary
The video discusses the cache stampede problem, where many requests hit a database simultaneously after a cache entry expires. It presents four strategies to mitigate this issue: cache locking, request coalescing, probabilistic early refresh, and background refresh, each with its own trade-offs.
Chapters
[00:02]
Cache Stampede Problem
When many requests hit the database at once after a cache miss, it's called a cache stampede. Most engineers only know one way to fix it, but there are four approaches.
[00:15]
Cache Locking
A lock allows one request to rebuild the cache while others wait. Simple to implement, but if the rebuild is slow or fails, requests may time out.
[00:31]
Request Coalescing
Duplicate in-flight requests are folded into one upstream call, and all get the same result. Cloudflare does this at the CDN layer, so even a million users hitting the same key results in at most one request per app server.
[00:44]
Probabilistic Early Refresh
As a cache entry ages toward its TTL, each request has a small but growing chance of triggering a background refresh. For example, at minute 50 of a 60-minute TTL, about 1% of requests refresh; at minute 59, about 20%. This spreads the refresh naturally.
[01:11]
Background Refresh
A dedicated worker proactively recomputes hot keys before they expire, eliminating stampede risk. The trade-off is needing to know what's hot ahead of time. Ticketmaster event pages are a perfect fit, but random traffic spikes are not.
[01:26]
Recommendation
Start with request coalescing as it's the easiest win and works at the CDN or app layer with no TTL changes. Add probabilistic early refresh for unpredictable spikes, and use background refresh for known hot keys.
The video provides a concise overview of four cache stampede mitigation techniques, recommending a layered approach starting with request coalescing and adding probabilistic or background refresh based on traffic predictability.
Study Flashcards (6)
What is a cache stampede?
easy
Click to reveal answer
What is a cache stampede?
easy
Click to reveal answer
A cache stampede occurs when many requests hit the database at once after a cache entry expires.
00:02
What is the downside of cache locking?
easy
Click to reveal answer
What is the downside of cache locking?
easy
Click to reveal answer
If the rebuild is slow or fails, requests may time out.
00:15
How does request coalescing work?
medium
Click to reveal answer
How does request coalescing work?
medium
Click to reveal answer
Duplicate in-flight requests are folded into one upstream call, and everyone gets the same result back.
00:31
What is probabilistic early refresh?
medium
Click to reveal answer
What is probabilistic early refresh?
medium
Click to reveal answer
As a cache entry ages toward its TTL, each request has a small but growing chance of triggering a background refresh.
00:44
What is the trade-off of background refresh?
medium
Click to reveal answer
What is the trade-off of background refresh?
medium
Click to reveal answer
You need to know what's hot ahead of time.
01:11
Which approach is recommended as the easiest win?
easy
Click to reveal answer
Which approach is recommended as the easiest win?
easy
Click to reveal answer
Request coalescing.
01:26
💡 Key Takeaways
📊
Request Coalescing at CDN
Cloudflare uses this at the CDN layer, showing real-world application.
00:31
🔧
Probabilistic Early Refresh Example
Provides concrete numbers (1% at minute 50, 20% at minute 59) to illustrate the concept.
00:44
💡
Background Refresh Use Case
Ticketmaster event pages are a perfect fit, highlighting practical applicability.
01:11Full Transcript
[00:02] requests hit your database at once. That's a cash stampede and most engineers only know one way to fix it. Let's go through four approaches. First, Let's go through four approaches. First, cash locking. A lock lets one request
[00:15] rebuild the cash while everyone else waits in line. Simple to implement, in your system. The downside of this is that if the rebuild is slow or fails, are going to be timing out. Another approach is to collapse requests instead
[00:31] of queuing them. Every duplicate in-flight request gets folded into one upstream call and everyone gets the same result back. Cloudflare does this at the CDN layer. Even if a million users hit the same key, your back-end sees at most
[00:44] one request per app server. We can also rely on randomness. As a cash entry ages rely on randomness. As a cash entry ages toward its TTL, each request has a small but growing chance of triggering a background refresh. So, at minute 50 of
[00:57] background refresh. So, at minute 50 of a 60-minute TTL, maybe 1% of requests refresh it. At minute 59, maybe 20%. Refresh is spread out naturally. The last is a background refresh. A dedicated worker proactively recomputes
[01:11] hot keys before they expire. This means zero stampede risk. The trade-off is you need to know what's hot ahead of time. Ticketmaster event pages are a perfect fit, but random traffic spikes are not. So, start with request coalescing. It's
[01:26] the easiest win and works at the CDN or app layer with no TTL changes. You can add a probabilistic early refresh for unpredictable spikes. Use background demand. Like and follow us for more system design tips.
