The sticky note on the counter
Every trick so far made the notebook faster to consult. Caching asks a better question: why consult it at all?
If forty customers an hour ask today's specials, the worker does not look it up forty times. They write it on a sticky note and stick it to the counter. Answering becomes glancing. That's a cache: a small, fast, temporary copy of an answer you keep having to produce.
The numbers are not subtle. Reading from memory is roughly a hundred times faster than reading from disk, and a cache hit skips the query, the index walk, and the network round trip all at once. Caching is the highest-leverage thing in this module - which is exactly why its failure modes deserve respect.
Imagine it like this: A sticky note with today's specials. Fast to read, occasionally wrong, and someone has to remember to update it.
- cache
- - a fast temporary store holding answers you'd otherwise recompute
- cache hit
- - the answer was already there - no database work needed
- cache miss
- - it wasn't there, so you pay the full cost and usually store the result
Cache-aside, and why it's the default
The common pattern is cache-aside: look in the cache; if it's there, return it; if not, ask the database, put the answer in the cache, and return it. The application owns the logic, which is precisely why it's popular - nothing magic happens behind your back.
The alternative, write-through, writes to the cache and the database together on every write. The cache is never stale, but every write pays both costs, and you fill the cache with things nobody may ever read.
Cache-aside wins by default because it only ever caches things somebody actually asked for.
def get_profile(user_id):
key = f"profile:{user_id}"
cached = cache.get(key)
if cached is not None:
return cached # hit: ~0.2ms
profile = db.query( # miss: ~15ms
"SELECT * FROM users WHERE id = %s", user_id
)
cache.set(key, profile, ttl=300) # remember for 5 minutes
return profilefirst call miss 15.2 ms next 4,999 hit 0.2 ms -------------------------------- average 0.203 ms (vs 15 ms uncached)
- cache-aside
- - the app checks the cache, falls back to the database, then stores the result
- write-through
- - every write updates cache and database together
- TTL
- - time to live - how long an entry may be served before it's discarded
Invalidation, and the two hard problems
There is an old joke that the two hard problems in computer science are cache invalidation, naming things, and off-by-one errors. The joke survives because invalidation genuinely is hard: the moment data changes, every copy of the old answer everywhere is now a lie, and nothing announces it.
You have two honest options. Expire on time - a TTL, which is simple and self-healing but means accepting a known window of staleness. Or expire on event - delete the key when the underlying row changes, which is fresh but requires you to find every affected key, and forgetting one means serving wrong data indefinitely.
Most systems use both: delete on write for the keys you can name, and a TTL underneath as the safety net for the ones you forgot.
Ten thousand requests a second are being served from one cached key. It expires. All ten thousand miss simultaneously, and all ten thousand hit the database with the same query - a database sized for a trickle, suddenly taking the full flood. Cold caches are how sites die right after a deploy or a restart. Defences: stagger TTLs with jitter, let just one request rebuild while others serve the stale value, or refresh popular keys before they expire.
- TTL - simple, self-healing, accepts a known staleness window
- Delete-on-write - fresh, but you must find every affected key
- Use both: event invalidation, with a TTL as the safety net
- Never let the whole cache expire at the same instant - add jitter
- invalidation
- - removing or refreshing cached data once it stops being true
- thundering herd
- - many simultaneous misses on the same key hitting the database at once
- jitter
- - randomising expiry times so entries don't all die together
Where the module leaves you
Four lessons, one thread. Indexes made each question cheap. Replicas spread the reading. Shards split the writing. Caches skipped the question entirely. Every one of them bought performance with a currency: write speed, freshness, flexibility, correctness under change.
That is the honest shape of state. There is no arrangement that is fast, always fresh, and simple. When someone asks how you'd scale a system, the answer they're listening for is not a list of technologies - it's which of those currencies you chose to spend, and why it was the right one for this product.
Module 3 takes the opposite approach: instead of making the work faster, we take it off the request path entirely.