Caching Explained: Redis, CDN, Patterns — Full Breakdown & Transcript

[00:02] In every app, there is a page like this one loaded over and over all day. Every load asks the database for the same answer. The database does the same lookup and builds the same reply over

[00:15] lookup and builds the same reply over and over. Nothing about that answer it changed in between. So, you keep a copy of the finished answer closer by and the next request stops there instead. That copy is a cache and the database

[00:33] goes quiet. What you just built is a second copy of your data and everything in front of it quietly starts leaning on that copy. By the end, you will know

[00:46] what's inside the box, where a copy can live, which pattern fits, how caches fail, what one costs, and when not to add one. Let's open the box. Every cash

[01:01] add one. Let's open the box. Every cash starts with a bet. A cash pays only if starts with a bet. A cash pays only if the same data is asked for again before the same data is asked for again before it changes. That's winnable because real

[01:13] it changes. That's winnable because real traffic is nothing like even. Facebook found the same shape in every workload they measured. A few keys carry most of the requests. have the keys in their pool barely show

[01:27] have the keys in their pool barely show up at all and the other half were popular enough to answer most of the traffic. Twitter found the same curve across its production clusters. The share of requests the shelf answers is

[01:43] share of requests the shelf answers is its hit rate and the rest is the miss its hit rate and the rest is the miss ratio. So the shelf earns its space and that miss ratio is the number your database fields. A second condition

[01:57] database fields. A second condition hides in the phrase before it changes. The page we will follow is the Aurora desk lamp. Read constantly and change it almost never. So what is the shelf made of? Every diagram draws the cache as a

[02:14] rectangle with a label on it. And what's inside is the least mysterious part of inside is the least mysterious part of the system. It's a hash table. Reddus the system. It's a hash table. Reddus describe its own as auto resizing power

[02:28] describe its own as auto resizing power of two sizes with collision handled by chaining. That is why a cache read is constant time instead of a search. The interesting part is the growth because moving every key at once will store the

[02:44] cache long enough to time out every client. So Reddus keeps two tables and migrates a few buckets at a time during ordinary operations.

[02:57] ordinary operations. New keys go to the new table. Lookups check both and the server never stops serving. The capacity that buys isn't theoretical. One instance holds billions of keys and was tested past a quarter of

[03:15] billion. Per key overhead is about 185 bytes. So the table is finished and something has to decide what will leaves. Ask most

[03:28] engineers what happens when a cache fills up and they will say it evicts the fills up and they will say it evicts the latest recent user keys. Reddus doesn't run that algorithm. It samples a few keys at random and effects the one idle

[03:45] longest. There is no global list anywhere in the system. The small number is five and it is a config value you can turn up. Three is faster and less

[03:58] turn up. Three is faster and less accurate. 10 gets close to true LRU and 64 is the selling. Now the part almost everyone has Now the part almost everyone has backward and I did too until I read the

[04:12] backward and I did too until I read the source. Reddus estimates instead to save memory rather than CPU because the true LRU costs more memory to track. Sampling

[04:25] alone would be too rough. So there is a short list of good evction candidates. A short list of good evction candidates. A sampled key only gets in if it beats something already there and the best of that pool dies. So five keys get

[04:41] sampled. A scoreboard keeps the best of them and an older key can sit there in them and an older key can sit there in sampled. If evctionection is a choice, sampled. If evctionection is a choice, the policy behind it is a menu. Sampling

[04:55] sounds like it loses something, but Reddus simulates it against longtail Reddus simulates it against longtail traffic and found the gap from true LRU traffic and found the gap from true LRU small or gone entirely. Turn the dial to

[05:10] small or gone entirely. Turn the dial to LFU. The counter is an estimate only LFU. The counter is an estimate only eight bytes wide. So a key hit 10 million times and one hit 2 million can hold the same number. By default, the

[05:25] counter tops out around a million requests and drops back down every minute. Frequency here means recent frequency. LFU samples the same way

[05:37] frequency. LFU samples the same way through the same five keys and 16 slots through the same five keys and 16 slots with only the score changing. Reddus 8.6 added a third LRM. Now the finding that should change what

[05:53] Now the finding that should change what you pick. Twitter reported that at you pick. Twitter reported that at reasonable cash size FIFO performs about reasonable cash size FIFO performs about as well as LRU and 76% of their cash

[06:06] as well as LRU and 76% of their cash were past that point. LRU pushes out object with long gaps between accesses and FIFO protects them. All of this and FIFO protects them. All of this assumes a full cache when most keys

[06:21] should leave on their own. The model most of us carry is that a TTL is a timer and when it fires the key is deleted. There is no timer. Reddus

[06:34] deleted. There is no timer. Reddus removes expiry keys two ways and neither removes expiry keys two ways and neither run at expiry. The passive one cleans up when a client asks for a key that already timed out and that read was

[06:48] correct. The second is active. A sampling loop 10 times a second. Test 20 random keys with an expiry. Delete the dead ones and if more than a quarter

[07:01] were dead, go again. Reddus calls it a trivial probabilistic algorithm that assumes the sample represents the key space. The 20 isn't even setible because

[07:14] space. The 20 isn't even setible because it is a compile time consistent which guarantees expired data hangs around. So Reddus publishes a bound at any moment. The number of already expired keys still in memory is at most

[07:32] expired keys still in memory is at most your peak right rate divided by four and those keys get counted. So database size doesn't tell you how many live keys you doesn't tell you how many live keys you have. Reddus samples to expire while

[07:46] have. Reddus samples to expire while mimach has a different problem entirely. Here's the failure that looks impossible from outside the box. Your cache reports from outside the box. Your cache reports free memory and refuses the right and

[08:00] free memory and refuses the right and both are true at once. Mim cached breaks both are true at once. Mim cached breaks the storage into one megabyte page each the storage into one megabyte page each assigned to a class holding items of one

[08:12] size band. Once a page belongs to a class, it is never moved. The doc says meme cache is many smaller individual meme cache is many smaller individual caches. each class with its own counters

[08:26] caches. each class with its own counters and its own LRU. So evacuating from one does nothing to another. That's lab classification. All your memory ends up in 100 byte items. The application starts sorting 200 byte items and those

[08:44] starts sorting 200 byte items and those chunks cannot satisfy them. The fix chunks cannot satisfy them. The fix shipped as opt-in flags in 1.4.11 4.11 shipped as opt-in flags in 1.4.11 4.11 and become the default in 1.5.0

[08:56] in July 2017 as automated slab rebalancing and it's not free. Reassignment evacuates every item on the page it moves. Both engineers do this on

[09:09] every few threads which sound like it should be the bottleneck. There are two should be the bottleneck. There are two opposite beliefs here and approves both wrong. One says single threading makes it fast and the other say it limits it.

[09:24] Reddus execute commands on one thread and says it is not designed to benefit from multiple cores. So if you need more, you run more instances, but the

[09:37] more, you run more instances, but the CPU rarely runs out. Reddus says it's usually memory or network bound and the benchmark docs put the limit at the network well before the CPU. The published numbers come from

[09:52] The published numbers come from different machines. The faster one 10 times the throughput from patching round trips was measured on a MacBook Air. trips was measured on a MacBook Air. Redest 6.0 zero added IO threads and

[10:06] what they thread is a client socket reads and writes with command execution reads and writes with command execution still on one thread it's off by default mim cache is multi-threaded and still hit its selling in a lock a system has

[10:22] room for a copy at every layer three questions this act just answered what's your evacuation policy naming a policy and why it fits your access pattern is

[10:35] and why it fits your access pattern is engineering. Answering LRU describes something Reddus doesn't run. What happens when a key expires? happens when a key expires? Expiry is lazy and sampled. So your key

[10:49] count and memory both lag what's logically true. Why is the cache slow? The useful answer almost never involves CPU, but round trips, value sizes, or

[11:03] instance count. Nobody publishes what interviewers score, but each of those interviewers score, but each of those has a real mechanism underneath. That's one cache in one box. A request for that product page can meet at least six of

[11:19] product page can meet at least six of them on the way in in order. The rule that makes them make sense is that the cheapest request never travels. Every cheapest request never travels. Every door that opens in the journey and the

[11:33] rest of the hallway never runs. The layers are in ranking by speed. They layers are in ranking by speed. They differ on how traffic they absorb, how personal their content can be, and how hard they are to correct when wrong. And

[11:49] it's measurable because the vendors publish it. Pastley states a global Berg inside 150 milliseconds. And the Cloudflare has measured its own median

[12:01] Cloudflare has measured its own median at about the same. Fastly grounds that at about the same. Fastly grounds that in something you cannot engineer around. in something you cannot engineer around. The fastest possible B is about 65

[12:13] milliseconds the time flight takes to cross the planet. Fastly put the exception in one sentence. CDNs cannot invalidate an end user's web browser

[12:26] cache. And which door opens isn't the browser decision. It's written in the browser decision. It's written in the response. The instructions that object carries are a handful of words in one header and two of them mean close to the

[12:42] opposite of what they say. No cache does not prevent caching. RFC makes it a restriction on reuse. So the copy sits in the cache and has to be validated

[12:55] in the cache and has to be validated before it's served which is a genuinely unfortunate piece of naming. No store is the one that forbids storage and it applies to private and shared caches alike. You can see it in how the RFC is

[13:13] written. The condition for storing require that no store is absent and no cache appears only in the list of reusing. Which layer may hold it comes down to one rule. Read it in order. Where smax

[13:31] one rule. Read it in order. Where smax age wins for shared caches. So one response can tell a CDN to hold it for a day and a browser for a minute. private day and a browser for a minute. private says a shared cache must not store this

[13:46] and the RFC notes it cannot ensure privacy. Storing it is one decision and checking whether it's still good is another. A copy that's no longer fresh

[13:59] isn't automatically wrong. It's unverified. So the cheapest next move is a yes or no question. Instead of refetching everything on revalidation, the cache sends the entity tag it was given. If

[14:16] sends the entity tag it was given. If nothing changed, back comes a 304 with nothing changed, back comes a 304 with nobody and the cache reuses what it has. nobody and the cache reuses what it has. A 304 saves the body and it does not

[14:29] save the rounded trip which on a slow connection is most of what the user waits for. Immutable removes the request entirely. It says the origin won't

[14:41] change this file while it's fresh. So clients shouldn't send a conditional clients shouldn't send a conditional request at all. It only holds while the response is fresh. Once the response goes stale, it gets revalidated normally

[14:57] as if immutable were never there, which is the pair you want on files with a is the pair you want on files with a hash in the name. A year of freshness plus immutable all of which assumes you can reach the origin to ask the copy is

[15:14] can reach the origin to ask the copy is still in the original cannot be reached. still in the original cannot be reached. HTTP has had two standard answers since HTTP has had two standard answers since 2010 that most engineers have never

[15:26] 2010 that most engineers have never used. The first one hides latency. Still while revalidate lets a cache return a stale response immediately while it revalidates in the background so nobody waits. The second one buys availability.

[15:43] Stale if error lets a cache return a stale response when it hits a 500, 52, stale response when it hits a 500, 52, 500 3 or 54 rather than passing a hard

[15:56] 500 3 or 54 rather than passing a hard error through. Then the cache has to stop and pass the error through. The window counts from the end of freshness, not from the request. And that's the part people get backward. must

[16:10] part people get backward. must revalidate says a stale response cannot revalidate says a stale response cannot be reused until it's validated. And if the origin is unreachable, the cache must error instead, which the RFC says

[16:25] must error instead, which the RFC says should be a 504. Use it only when a stale answer would break something. And there is one layer you can never reach. there is one layer you can never reach. Every layer so far has an operator. and

[16:39] an operator has a bark button. The browser cache has neither and it sits on hardware you don't own. There is no mechanism for it. And once you have seen

[16:51] that the browser DTL stops being a freshness setting and becomes a freshness setting and becomes a commitment. AWS prefers versioned file names over invalidation for the same reason. Versioning controls which file a

[17:07] request returns even when the user already has one cache. Invalidation on the layers you don't control isn't free either. And a wild card counts as one

[17:20] path even when it clears thousands of files. MDN calls this cache busting. Put a hash of the content in the file name. So a changed file gets a new URL and

[17:34] there is nothing left to invalidate. For URLs that can genuinely change. Fastly recommends caching long at the edge with surrogate control and telling

[17:47] edge with surrogate control and telling browsers not to store it. And that layer browsers not to store it. And that layer isn't one cache. The CDN has been one isn't one cache. The CDN has been one door in this hallway and it is not one

[18:00] door in this hallway and it is not one of anything. Every city Cloudflare lists is a separate cache holding its own copy. Viewers in different regions route through different edge caches and each can ask your origin for the same

[18:16] content. So the CDN having it and not having it are both true at once. which took me a while to get my heads around. The fix is to stack them. Cloudflare's

[18:29] tiered cache splits their data centers into lower and upper tiers and on a miss only the upper tier may ask the origin. The measured payoff comes from Argo as a

[18:42] The measured payoff comes from Argo as a whole rather than tiered caching alone. whole rather than tiered caching alone. a 60% decrease in cash misses and it is a 60% decrease in cash misses and it is not a free win. A cami documents as

[18:54] straight either or. A local map takes the most load of your origin. A global the most load of your origin. A global map gets the content closest to the user and one setup cannot do both. Every layer so far sits outside the database

[19:10] and there is a reason for that. Why not let the database cache its own query let the database cache its own query results? My SQL shipped exactly that, run it for over a decade and then deleted it. The query cache was

[19:25] deleted it. The query cache was duplicated in 5.7 and remove it in 8.0 and my SQL's stated reason is scalability. It could easily become a

[19:37] server bottleneck. The mechanism that killed it is a trap you can build at any layer. One right to a table throws away every cached result that touched it

[19:49] every cached result that touched it affected rows or not. The reason almost nobody codes is the better one. Only queries that hit sees any improvement. So it never made the performance more predictable. Users upgrading were

[20:05] pointed at proxy SQL as a man in the middle cache. So a database vendor's answer to query caching was to move the cache out of the database. What survived

[20:18] caches pages rather than answers and one right invalidates only the page it touched. How much one right throws away is the entire difference.

[20:30] Three more from this act. Where would you put the cache? Name what you are optimizing for because answering Reddus picks a product, not a place. How

[20:43] do you invalidated? It differs by layer and the one nearest the user cannot be reached. So you change the URL instead. Why not cach it in the database? A buffer pool and a query cache are

[20:59] different things. and a right clears very different amounts of each. very different amounts of each. Every layer holds the same copy and disagrees independently. So placement decides how many truths you own. So how

[21:15] decides how many truths you own. So how does data get into the cache and out does data get into the cache and out again when it changes? Cash aside is the pattern almost everyone writes first and it removes the key on a right rather

[21:29] than updating it. Facebook settled that one. They delete rather than update one. They delete rather than update because deletes are item potent and a right carries a value that can arrive out of order. The second is ordering.

[21:45] update the store before removing the cache item because deleting first lets a client put the stale value straight back. Now the part that survive is doing

[21:57] both correctly. Facebook named it a stale set. A fill reads the old value. stale set. A fill reads the old value. The invalidation lands first and then the in-flight reply writes the old value back which leaves the new price in the

[22:13] back which leaves the new price in the database and the old one in the cache indefinitely because the next read is a hit. The lamp is on stale at 59 in the

[22:25] hit. The lamp is on stale at 59 in the database and 84 on the page and nothing in that code is a bug. That race needs something that can tell a late right it losts. Facebook built exactly that and almost

[22:40] Facebook built exactly that and almost nobody covers it. On a missachid nobody covers it. On a missachid hands the client a 64 byte token bound hands the client a 64 byte token bound to that key and a set must present it.

[22:52] If a delete arrived in the meantime the token is invalidated and the set is rejected. The read reserves the right to write and anything in between revokes it. So the stale set from a minute ago

[23:08] never lands. The late fill is turned away and both sides agree again. The same token solves a second problem for free. A server hands out a token only

[23:22] free. A server hands out a token only once every 10 seconds and per key. So everyone else waiting and only one client regenerates it which produces the number worth remembering from this act. Leases cut the peak database query rate

[23:39] Leases cut the peak database query rate by 13 times on keys that draw crowds. And since they provision on peak that's 13 times less hardware. Waiting isn't the only option since a get can return data market still for

[23:57] callers that can use it. The other family right to the cache cash aside deletes and accept a miss afterwards. So the obvious improvement

[24:10] is to put the new value straight into the cache that is right through sold as the cache that is right through sold as the pattern that removes tailness. Microsoft documenting its own right through architecture

[24:25] states the problem in one line. The design doesn't provide atomic design doesn't provide atomic distributed transactions across the two. Clipman's argument cover this directly.

[24:38] Since a cache is a duplicate of data in a database and two clients writing in different orders leave the two permanently inconsistent and the permanently inconsistent and the instinct here is to retry which is where

[24:52] I would have gone too. Conflant puts that to bed. An in-memory retry dies that to bed. An in-memory retry dies with the process and a durable retry can fail independently. AWS says the two fail in opposite

[25:08] direction. Cash aside allows a stale data but survives empty nodes while write through fills the cache with things nobody reads. So write through

[25:20] moves to the failure rather than removing it. And if two rightes cannot be atomic, the invalidation has to travel inside the right. So right through moves the failure rather than removing it. And if two rightes cannot

[25:37] be atomic, the invalidation has to travel inside the right. There are two production answers. The first is a transactional outbox. record the cache

[25:49] update intent in the same transaction as the business right so it survives with the data that is the distinction that matters since it sounds like a durable

[26:01] matters since it sounds like a durable retry we rule it out this one lives or die with the business right and a repeated invalidation it changes nothing it does tie you to your own code your application has to remember to write the

[26:17] So anything writing from outside your code invalidates nothing. The second derive the invalidation from what the database already committed. Facebook run

[26:30] database already committed. Facebook run this in 2013 with a demon extracting deletes from the statements each database commits and those logs mean a database commits and those logs mean a lost invalidation can be replied. Uber's

[26:44] lost invalidation can be replied. Uber's cache front tails my SQL bin logs and invalidate Reddus in under a second serving serving 150 million reads a second. Uber also

[26:57] writes timestamped invalidation markers a tombstone that refuses anything older a tombstone that refuses anything older than itself. Right behind is usually introduced as the first one. The right lands in the cache the client is told it

[27:13] has done and the database is updated later from a queue. Oracle's own later from a queue. Oracle's own sentence right behind effectively makes the cache the system of record until the queue reaches the desk. Your durability

[27:29] guarantee is now whatever your cash tiers a replication provides. The transaction side gets discussed even less. The cache transaction completes

[27:42] before the database transaction begins which means it must never fail. You already said it worked it. Replicating the queue narrows the window without removing it. Puzzle cast keeps dirty entries on primary and backup. So

[27:59] durability becomes the cash tiers replication guarantee. So right behind actually ask whether you can lose the last few seconds of confirmed rights

[28:11] when a node dies which is fine for a view counter and not for a balance. The queue also merges updates. So a 100 updates to one key becomes one right and

[28:25] the tidy table those four patterns come in doesn't survive checking right around in doesn't survive checking right around has no primary definition anywhere nine primary sources and it appears in none of them which I did not expect when I

[28:41] went looking. Search for caching patterns and you will find the same tidy table everywhere with confident definitions and no sources. That is one of two things about it that don't survive checking. Three major vendors

[28:57] define right through three different ways and they don't agree. Oracle has the application right to the cache and wait for the data source while AWS and

[29:10] wait for the data source while AWS and Microsoft write the database first and Microsoft write the database first and it is just awarding a problem. AWS lists it is just awarding a problem. AWS lists as an advantage that cached data is

[29:22] never stale while Microsoft chips a repair function because the reddest repair function because the reddest write can fail after the commit. Refresh write can fail after the commit. Refresh ahead has exactly one precise definition

[29:35] ahead has exactly one precise definition and its oracles which also names the failure the explainers leave out. A wrong guess send pointless requests to the database. Read through just means the cache lands on a miss instead of

[29:51] your code. So describe the mechanics not the table. And what actually keeps the table. And what actually keeps stillnesses in check in most systems is stillnesses in check in most systems is much simpler. Those are all real and

[30:05] almost nobody has them. What nearly every production system relies on to keep staleness in check is a number somebody typed once, usually around one.

[30:19] And the tradeoff is a curve you can compute. The hutnet's model states the boundary plainly. As the TTL goes to zero, the miss ratio goes to one. So

[30:32] drive it down and you have paid for a cache that never hits. What do real system use? 2/3 of Twitter's clusters sit at 12 hours or

[30:44] less and write heavy workloads run far and write heavy workloads run far shorter than read heavy. Twitter's soft TTL stores a shorter TTL inside the value and when that expires it refreshes

[30:59] in the background while still serving the older one. It is a stale while revalidate inside the value and the selling all of this is chasing meta measures cash consistency in nines and

[31:15] went from 6 to 10. Define it as consistent within 5 minutes. Consistency is measurable and even with meta's money behind it, the answer isn't 100%.

[31:29] And most systems run more than one cache. Three from the patterns act. How do you keep the cache and the database in sync? Name the mechanism abounding the disagreement because sync itself isn't achievable. Do you update the

[31:46] isn't achievable. Do you update the cache on right or delete the key? Delete because deletes are item potent and values can arrive out of order. Which caching pattern would you use? describe the mechanics rather than naming one

[32:02] because the three vendors define right through three different ways. Every pattern answers one question. What happens between the truth changing and the copy catching up? So what changes when there is more than one cache? Large

[32:19] systems run two a small one inside each process and the shared remote one behind it. The reason to reach for it is in speed. Microsoft stated benefit is

[32:32] availability. Processes holding a local copy keep serving when the shared cache is unreachable. Netflix ships this inside their client

[32:44] where a flag turns a one tier cache into a two-tier one without touching application code. The price is that the copies drift apart. Each application

[32:57] copies drift apart. Each application instance has its own independent cache. instance has its own independent cache. So 10 processes means 10 snapshot taken at different moments. And Huzzlecast says near cache breaks a strong

[33:10] consistency and the invalidation meant to fix that is patched and losy by design. Hazlecast patches them. pillar rate misses and runs a clean up swip

[33:23] which exists because the main path drops things. A filed huzzlecast bug shows the cost on rarely updated keys. The drift swip never fires so those inteitely.

[33:41] Netflix simply promises nothing and the remote cache is itself several machines which raises the only question that matters about a cache cluster given a matters about a cache cluster given a key which machine holds it hash the key

[33:55] module the machine count and that works until you add one let's do the arithmetic because nobody publishes it a key stays put only if it's hash module n and module n + one agree. So the

[34:11] and module n + one agree. So the fraction that moves is roughly n / n + one. And in a cache, nothing moves them. So every relocated key is just a miss So every relocated key is just a miss all at once. Lipkitama author says it

[34:25] whippid the entire cache. Consistent hashing exists to make that fraction hashing exists to make that fraction small. The 1997 paper calls it smoothness and monotonically which means keys only ever move toward the new node.

[34:41] keys only ever move toward the new node. The mechanics is a ring. Liptama places each server at many points on a circle. A key walks to the next point and those A key walks to the next point and those many points keep the arcs even. Reddus

[34:55] cluster uses fixed hash slots instead and meme cached keeps the map of servers in the client. What happens to key during a move isn't settled. There is an

[35:07] during a move isn't settled. There is an obvious assumption here and it is wrong. obvious assumption here and it is wrong. Cashed data can always be rebuilt. So Cashed data can always be rebuilt. So surely a cash drops it and refills.

[35:19] surely a cash drops it and refills. One major system does that and the other One major system does that and the other doesn't. Redisk cluster moves the data. doesn't. Redisk cluster moves the data. A hash slot is just a set of keys. So

[35:31] resharding moves keys between instances and outside of resharding data never moves at all. The interesting part is a request The interesting part is a request arriving during the move. Redes cluster

[35:46] arriving during the move. Redes cluster uses a oneshot ask redirect sending only uses a oneshot ask redirect sending only the next query to the new node. So both the next query to the new node. So both stay reachable until it completes

[35:58] and only the next query is the elegant bite because a permanent redirect would be wrong while the slot is mostly on the old node. Mim cached has no position to

[36:12] old node. Mim cached has no position to take. It provides no server to server coordination. So nothing could move a key since no So nothing could move a key since no server knows any other exists.

[36:25] Which one you want depends on what your origin does when a slice of your cache disappear. Everything so far has been the cache working. Starting from here, it fails. One popular key expires while

[36:41] many requests are in flight. and every one of them rebuilds the same value. That's a cash stamp. It also worsen itself. The paper calls it a cascading

[36:54] itself. The paper calls it a cascading failure because several rebuilds at once slow each other down which widen the window which pulls in more requests. The window which pulls in more requests. The fix most people reach for is a lock and

[37:08] it is the worst of the three options with four documented costs including with four documented costs including nothing being served if the holder dies. nothing being served if the holder dies. The optimal fix is one line rebuild

[37:21] early by a gap. That's an exponential draw scaled by how long the last rebuild took. So it needs no knowledge of your request rate. And I will admit it. I

[37:34] reread that line three times before it clicked. It calibrate itself too because the gap is multiplied by the measured rebuild time. So expensive values

[37:47] refresh earlier and the cheap ones later with nothing configured. It's also the best you can do without knowing your request rate. And the plain even randomness is probably worse. The opposite case is a key that doesn't

[38:04] exist. Every failure so far has been the cash holding the wrong thing. This one is the cache working perfectly and giving zero protection because you cannot cach the absence of something you never looked

[38:18] up. The request misses the origin returns nothing and nothing is what gets cached forever. And because the key is attacker chosen, this is the one failure

[38:30] somebody can aim at you deliberately. The cheap fix is to cash the miss with a different TTL than positive entries because a negative entry goes stale the

[38:43] instant somebody creates the thing. Negative caching stops the same key repeating and does nothing about someone walking through thousands of new ones. walking through thousands of new ones. The other fix answers a harder question.

[38:57] Can you know a key is absent without asking? A bloom filter can. And 1% costs roughly 10 bytes per item at any scale. The cost depends only on the error rate. A no is always right while a yes is a

[39:14] A no is always right while a yes is a maybe. You cannot remove an item once it maybe. You cannot remove an item once it is in although a new variant can. Next is a key everybody wants on one machine. The flash sale is live. The lamp is the

[39:29] most requested item on the site and one machine is on fire while your hit rate machine is on fire while your hit rate stays excellent. A key maps to exactly

[39:41] one slot served by a single node. And that predictability is what makes lookup works without coordination and what concentrates a hotkey. Adding nodes move

[39:53] the slot without ever splitting it because a key needs exactly one home for because a key needs exactly one home for the routing to work. Adding hardware cannot fix this. The same physics show up whenever a partition has a fixed

[40:09] up whenever a partition has a fixed selling. Discord documented because every read has to agree across several copies. Unrelated to traffic suffers copies. Unrelated to traffic suffers too, which is why the dashboard won't

[40:23] too, which is why the dashboard won't warn you. Your hit rate on that key is basically 100%. And the machine is still saturated. And the machine is still saturated. Reddus ships hot keys in the CLI which

[40:38] Reddus ships hot keys in the CLI which only works under LFU. If more machines cannot help, the only thing left to change is the key. First, the fix that change is the key. First, the fix that isn't one. Facebook rejects resharding a

[40:52] dead node's key onto survivors because whoever inherits the hot key falls over whoever inherits the hot key falls over next. Pix one is to split the key. Dynamo DB appends a random suffix spreading one key across many

[41:09] destination and the bill is that reading the whole key means querying every the whole key means querying every suffix. Fix two is to cache it in the process. Reddus ships a clientside caching where the server pushes

[41:25] invalidation to connection that have read a key on frequently updated data. Those messages costs more [snorts] than they save. Pix 3 is to merge the

[41:38] requests. Discord queries the database once when several user want the same row and the bill is coupling since every subscriber shares one request fate. A

[41:52] hotkey is a routing problem. So every real fix changes routing. Splitting changes the destination. Clientside caching remove the request from the caching remove the request from the network and merging turns many request

[42:07] network and merging turns many request into one. Next is every key expiring in the same second. This is the same failure at population scale and it comes from something innocent. Keys written together expires together. It's also a

[42:23] failure inside the engine because Reddus's own latency documentation says many keys expiring in the same second can make it block. The fix is to

[42:36] second can make it block. The fix is to stop them lining up. AWS says to add stop them lining up. AWS says to add jetdder to your TTLs and publishes no jetdder to your TTLs and publishes no recommended range. So that value is one

[42:48] you own and jetter isn't universally correct. GitHub. It changed a refresh DTL on a cache of user settings from 12 hours to two deployed on a Saturday and

[43:02] nothing broke until Monday when the extra load overwhelmed the database behind authentication. GitHub names a second factor. The cachet payload had quietly grown from bytes to kilobytes

[43:19] and the long TTL had been hiding it. So a TTL is a capacity setting for your original. Next is the cache simply not being there. Every failure so far was the cache doing something wrong. This

[43:35] one is it doing nothing at all and the load it absorbs had to go somewhere. Wik load it absorbs had to go somewhere. Wik Media has the one public postmortem with Media has the one public postmortem with the numbers. Their hit ratio fell while

[43:49] demand stayed flat which put five times the traffic on the origin and it recovered on its own. Cloudflare measured the other direction. When the storage behind their cache failed, the request that survive it were the ones it

[44:06] could answer. So your hit ratio is your blast radius number. A cash can become the weapon. Facebook run a job that replaced any cash config value it judge

[44:19] replaced any cash config value it judge it invalid. So when the database errored every client deleted the key instead. A repair loop with no rate limit is a denial of service tool you built and pointed at yourself. The dependency can

[44:35] also be one machine with app servers freezing on it rather than slowing down. freezing on it rather than slowing down. Honeycom's schema cache was kept warm by Honeycom's schema cache was kept warm by ingest traffic. So when injust stopped

[44:50] ingest traffic. So when injust stopped the reads overloaded the database. AWS says the cause behind this is usually organizational not technical. The fix is provisioned rather than clever and starting without a warm cache is its own

[45:06] failure. So what would your code do when the cache stops answering? Without a timeout, it waits forever on a dead store and everything behind it stops. A

[45:18] timeout saves the request, but each one pays the weight and then hits the database. Microsoft tells you to fall back to the data store and warns the back to the data store and warns the fall back can swap it. The third version

[45:33] stops calling the dead node altogether. A breaker opens a capid number reach the database and the rest are turned away at the door. None of this is atomic. So the

[45:46] blast radius lands on whoever owns the fallback. That is a dead cash handled well. An empty one is a different problem. Starting without a cache sounds

[45:58] gentler than losing one, and it's often worse. Google SR ebook is blunt about worse. Google SR ebook is blunt about the risk of outages under a cold cache. Their fix is a slow ramp instead of flipping it on all at once. increasing

[46:16] load slowly. So a small request rate warms the cache and a standby with no traffic has a cold cache. Warming isn't fast at scale since large

[46:28] Warming isn't fast at scale since large caches can take days. Facebook built caches can take days. Facebook built cold cluster ramp up so a cold cluster reads from a warm one and returns in hours.

[46:40] Roblox is what happens when the cold path is untested. Their caching tier had to be rebuilt from scratch because their tooling assumed deployments already

[46:53] handling traffic. Slack's version is the one where nothing Slack's version is the one where nothing failed at all. Called the client caches on the first Monday back pulled more data than usual, hitting gateways that

[47:07] hadn't scaled. Honeyump's trap is the dangerous one. The recovery path needed the cash already worn. So the only way out was a sequence nobody had rehearsed.

[47:21] One config line can stop a cache too. The named failure modes are over. This one is a single config word. The volatile policies behave like noction.

[47:34] volatile policies behave like noction. If no keys have an expiration volatile LRU sounds like the caution choice and it is but if nothing carries choice and it is but if nothing carries a TTL there is nothing it allowed to

[47:48] remove. So the cache cannot free anything under no eviction reddus rejects rightes with an out ofmemory error while reads keep working. So your

[48:01] dashboards show a healthy cache while half your application is broken. It only appears once the instance fills. The fix appears once the instance fills. The fix is one word all keys LRU which is also

[48:17] more memory efficient because it doesn't need to store an expiry bear key. The second trap is memory that never comes back. back. Reddus won't always return memory to the

[48:32] Reddus won't always return memory to the operating system when keys are removed because the allocator holds it for the reuse. So you provision for peak. Active def fragmentation exists for that off by default and only worth it sometimes.

[48:50] And one oversized value can stop a cache too. Earlier I said single threading isn't the bottleneck and that's true for ordinary traffic. This is the case where

[49:04] one command's cost gets a charge to everybody. Reddus own docs warn about everybody. Reddus own docs warn about keys more than anything else and people still run it in production. The same page tells you a laptop can scan a

[49:21] page tells you a laptop can scan a million keys in 40 milliseconds. The problem isn't that it's 40 millisecond. The problem is that it is 40 millisecond during which nothing else is served. So at 10,000 a second 400

[49:39] is served. So at 10,000 a second 400 requests are sted behind command. Not requests are sted behind command. Not every O of N warning is equally true. every O of N warning is equally true. Since Lrange costs by distance from the

[49:51] nearest end, so the whole list call is dangerous and a bounded page is cheap. The size selling are real. Elastic cache won't migrate slots holding items above

[50:05] 256 mgabytes. So one oversized value can mgabytes. So one oversized value can freezes your clustered layout. And the worst moment is a TTL firing because Red Bus freezes that memory in one blocking

[50:22] Bus freezes that memory in one blocking step when the key expires. So the pose arrives with nobody having asked it. Next is a working cash handing the wrong Next is a working cash handing the wrong answer. Three from the failures act.

[50:36] What happens when the cash goes down? Failing back is the trap because that capacity was sized for postcache traffic. Server stail or fail over to a

[50:50] standby. How do you handle a hotkey? It's the one failure where adding It's the one failure where adding hardware does nothing. So every real fix it changes the key or the routing. What happens when a popular key expires? A

[51:05] happens when a popular key expires? A lock is the obvious answer and the worst lock is the obvious answer and the worst option. Caches fail in ways that look healthy from outside. So how does a cache know which thing you asked for? It

[51:19] resets on the cache key which is smaller than anyone thinks. the method and the than anyone thinks. the method and the URI. The same URL returns a different bytes depending on who's logged in, what language they asked for and which tenet

[51:35] language they asked for and which tenet they are in. None of which is in the key. Vary is the mechanism for widing it. A cache must not reuse a stored response unless the headers named by vary match. And in practice, CDN's don't

[51:53] honor it by default. So the header designed to protect the cache key is the one the caching layer skips. Cloudflare's default adds a specific set of headers. Exactly the one used in the 2018 cash poisoning research. So a

[52:11] vendor's default cache key records what went wrong before. Vendor defaults give you safety, you shouldn't mistake for the standby. Since RFC says a response

[52:23] with a cookie can be cached and somebody else opens your pigeon holes. else opens your pigeon holes. Steam Christmas Day 2015 under a denial of service attack, Valvi deployed a second caching configuration

[52:39] and it cached the traffic for authenticated users. So, store pages for authenticated users. So, store pages for 34,000 people went to the wrong people. Read the sequence back. A cache was added to survive an attack and the cash

[52:54] leaked the data. The attack didn't branch anything. The mitigation did. The same gap is a deliberate attack surface too. Uncked inputs are components that

[53:07] change the response but aren't in the cache key. So an attacker can store a cache key. So an attacker can store a hustle response for everyone. There is a version that needs no special headers at all. Web cache deception was

[53:23] demonstrated on PayPal where a stylesheet suffic made the cash store an stylesheet suffic made the cash store an authenticated page as a static asset and it is a blocking that get burned down because a second run 14 months later

[53:38] found a mostly different set of vulnerability sites. The fix is to put more into the key which RFC calls double keying. What

[53:50] remains is whether to add one at all. Everything up to here was mechanism and Everything up to here was mechanism and the rest is the decision. It start with arithmetic order and distributed caching. Access time equals hit time

[54:07] caching. Access time equals hit time plus miss rate times penetly. Miss rate plus miss rate times penetly. Miss rate on its own is misleading because it ignores what a miss costs. A miss coasting two milliseconds and one

[54:22] coasting two milliseconds and one coasting 200 look identical. Now the coasting 200 look identical. Now the part that isn't a straight line 95 to 99% takes the miss rate from 5% to 1 which

[54:37] is five times less traffic at the origin. Twitter reports miss ratio for origin. Twitter reports miss ratio for exactly that reason. Fastly says the

[54:49] same for CDN's. So the first change to make tomorrow is putting miss ratio on the dashboard. There is a second correction. The highest miss ratio decides how many requests a second the back end has to

[55:06] requests a second the back end has to serve. So your origin is sized by the serve. So your origin is sized by the worst minute rather than the average. And the finding almost nobody has heard a 99.99%

[55:19] cache means an origin never load tested at real traffic. So its size stops being a guess. How big should the cache be? Gets asked in every interview and

[55:31] Gets asked in every interview and answered in almost none. The tool is the answered in almost none. The tool is the mis rate you curve which shows how your mis rate you curve which shows how your miss rate falls as the cash grows. You

[55:43] read a size of it by walking right until the next gigabyte stops buying anything. The other number is your working set. The total size of every different object

[55:57] The total size of every different object touched in a window of time. Matson touched in a window of time. Matson worked this out in 1970, getting the whole curve from one pass over a trace instead of one simulation

[56:11] over a trace instead of one simulation per size. 56 years old and most teams still guess. Exact curves were too heavy for production until shards in 2015

[56:25] built them from under 1% of request in a megabyte of memory. With DTLS in play, the row working set buys far more cash than you will use because expiry makes

[56:40] the real set level off. size isn't the only lever or the cheapest since Meta gained percentage points of hit rate from the eviction

[56:52] policy alone. Whether it's worth paying for it is another question. Most caching advice treats cost as obvious. Memory is cheap. Databases are expensive. Therefore, cash

[57:09] expensive. Therefore, cash here are the actual prices. pulled it in August 2026. The myth dies there. A cache node is only about 21% cheaper than a comparable read replica

[57:26] per gigabyte. It flips because cache memory costs over 100 times database memory costs over 100 times database disk. A cache is only worth the traffic

[57:38] disk. A cache is only worth the traffic it absorbs. And I wish I would frame it it absorbs. And I wish I would frame it that way years earlier. So $160 that way years earlier. So $160 node at 20 cents per million IO breaks

[57:50] even at 3004 requests a second. That assumes one read requests a second. That assumes one read IO per request reaching the origin which

[58:02] is conservative. So the real tipping point sits lower. Valky is priced a flat point sits lower. Valky is priced a flat 20% below Reddus on the same hardware.

[58:14] So one line of config drops the break even to 243. Serverless has a trap at roughly five times the per gigabyte

[58:27] at roughly five times the per gigabyte rate of a node. So it wins only when rate of a node. So it wins only when data is small or traffic is spiky. And there are workloads where no price is worth it.

[58:40] worth it. The cold open it promises the scene. A cash only pays when the same data is asked for again before it changes. asked for again before it changes. First, no repetition to exploit it.

[58:54] Twitter measured it directly. The more often an item is written, the less often often an item is written, the less often it's read. And an item touched once it's read. And an item touched once gives you nothing to hold onto.

[59:08] gives you nothing to hold onto. But do not turn that into right heavy means no cash. A third of Twitter's cash clusters were A third of Twitter's cash clusters were right heavy and still worth running. So

[59:22] that's a warning to measure rather than a verdict. Second freshiness you cannot a verdict. Second freshiness you cannot compromise on. Microsoft position is that you design for eventual consistency and if your requirement forbids that the

[59:40] cache is the wrong tool. Third, what your origin does without it and the test your origin does without it and the test for that is one sentence in AWS's guidance. Run load tests with caches disabled.

[59:54] three questions and if you have never answered the third, you have added a dependency you haven't tested. One workload breaks this so badly that

[1:00:07] two databases built defensive. There is one workload that destroys the cache for everyone else while getting nothing itself and it is ordinary a scan larger

[1:00:20] itself and it is ordinary a scan larger than the cache. It has a name sequential flooding that turns LRU exactly backward because under a scan the most recently used it

[1:00:34] under a scan the most recently used it page is the one you are finished with page is the one you are finished with and the best one is to effect postgress fixes it in the engine with nothing to configure pages that should only by a

[1:00:49] big scan get a small ring of buffer ers to cycle through. Instead of blowing out to cycle through. Instead of blowing out the buffer cache, my SQL InoDB does it the buffer cache, my SQL InoDB does it differently with midpoint insertion. New

[1:01:04] differently with midpoint insertion. New pages go 38 of the way down the list instead of the top. So a scanned pages sit near the exit and hot ones survive. Two mature engines shipped different mechanisms for the same failure. So

[1:01:21] mechanisms for the same failure. So naive LRU plainly fails under a big naive LRU plainly fails under a big scan. Your Reddus has neither. So keep scan. Your Reddus has neither. So keep the scan out. Running one is a separate

[1:01:33] the scan out. Running one is a separate skill. One last thing and it catches teams who got everything else right. Start with what AWS says to alarm on

[1:01:45] because hit ratio is not on the list. Hit rate is a performance signal. The alarm list is for the things that pays you at 3 in the morning. Evction have a

[1:02:00] cost most teams miss because a high evction rate drives engine CPU up and that's the CPU serving your requests. And here's where the industry watches

[1:02:13] the wrong number. Reddus says the fragmentation ratio everybody blotss is in the metric to act on because the allocator ratio is the true one.

[1:02:26] Connections have a hard selling a closer than people think and a clamping count than people think and a clamping count on flat traffic means they are not being closed. And on the same screen, put or region

[1:02:40] And on the same screen, put or region load the only metric that tells you what the cash is worth. Once you have seen it as a second copy of the truth, you will recognize these questions everywhere. The last three, how big should the cash

[1:02:57] The last three, how big should the cash be? Name the working set. Read the size of a missio curve instead of picking it. And note that expiry shrinks it. Is it And note that expiry shrinks it. Is it worth it? A cash justifies itself on

[1:03:12] traffic absorbed. So the question is what rate it takes of the origin. When would you not use a cache? Three checks and the third is what your origin

[1:03:25] does without it. Every checkpoint has been the same move. name the mechanism underneath the question and answer from it. Here's the whole thing in one frame.

[1:03:38] it. Here's the whole thing in one frame. We opened the box, counted where a copy can live, and everything after was a question about that copy. When it's wrong, the copy drifts the moment the source changes. So delete rather than

[1:03:54] update and put the invalidation inside the transaction. When it's gone, origin capacity gets sized for the traffic that survives the sized for the traffic that survives the cache. So your hit rate is also the size

[1:04:09] cache. So your hit rate is also the size of your outage. When it's lopsided, one key hashes to one node. So adding machines relocates the problem. And when it's empty, a cold cache multiplies load at the worst moment.

[1:04:26] And when it costs more than it saves because cache memory is only modestly because cache memory is only modestly cheaper than a read replica. So absorbed traffic is what justifies it. There is one check underneath all of it that

[1:04:42] one check underneath all of it that almost nobody runs. Load test with cache disabled. And that's the system working. The page served from cache. The origin a The page served from cache. The origin a quiet behind it. Nothing on fire. Thanks

[1:04:57] for staying through a long one. Subscribe if this was useful. Like if it should reach someone else and share it with whoever on your team is about to with whoever on your team is about to add a cache.