Queues & async work

The customer shouldn't wait for the delivery van

A customer orders a sofa. The worker takes the order, takes the payment, hands over a receipt, and says 'it'll arrive Thursday'. What the worker does not do is ask the customer to stand at the counter until the van gets back.

Software forgets this constantly. Someone signs up, and the request handler creates the account, then sends the welcome email, then generates the avatar thumbnail, then syncs to the CRM, then finally returns the page. The user stares at a spinner for four seconds - and if the email provider is having a bad afternoon, the signup itself fails. Because of an email.

The fix is the receipt. Do the part the user is actually waiting for - create the account - and write the rest down as a job for someone else to pick up. That written-down job goes in a queue.

What the user feels

Moving four steps off the request path doesn't make them faster. The email still takes 800ms. But it stops the user paying for it, and it stops one flaky dependency from failing an unrelated action.

It also changes the failure story completely. Synchronously, a dead email provider means nobody can sign up. With a queue, the jobs pile up harmlessly and drain when the provider recovers. Users never notice.

# Before: the user waits for all of it
def signup(email, password):
    user = db.create_user(email, password)   #   40 ms
    send_welcome_email(user)                 #  800 ms  (and can fail!)
    generate_avatar(user)                    # 1200 ms
    sync_to_crm(user)                        #  900 ms
    return user                              # 2940 ms total

# After: the user waits for the part they asked for
def signup(email, password):
    user = db.create_user(email, password)   #   40 ms
    queue.push("welcome_email", user.id)     #    1 ms
    queue.push("generate_avatar", user.id)   #    1 ms
    queue.push("sync_to_crm", user.id)       #    1 ms
    return user                              #   43 ms total

before   2940 ms   fails if ANY of the four fails
after      43 ms   fails only if the account can't be created

Retries, backoff, and the dead-letter queue

Jobs fail. The provider times out, the row isn't there yet, the network hiccups. The point of a queue is that failure is survivable: the job goes back on and is tried again.

Retry naively, though, and you build a weapon. A struggling service that gets retried instantly, by every failed job, in a tight loop, is a service you have finished off yourself. The rule is exponential backoff - wait 1s, then 2s, then 4s, then 8s - plus a little randomness so ten thousand jobs don't all retry on the same tick.

And some jobs will never succeed. The email address is malformed; the record was deleted. Retrying forever means a poison job spinning until the end of time. After N attempts it goes to the dead-letter queue - a parking bay for jobs a human needs to look at. An empty DLQ means healthy; a growing one is your earliest warning that something is wrong.

import random

MAX_ATTEMPTS = 5

def delay_for(attempt):
    base = 2 ** attempt                    # 1, 2, 4, 8, 16 seconds
    return base + random.uniform(0, base)  # jitter: spread the retries out

for attempt in range(MAX_ATTEMPTS):
    print(f"attempt {attempt}: retry in {delay_for(attempt):.1f}s")
print("-> dead-letter queue (a human should look at this)")

attempt 0: retry in 1.4s
attempt 1: retry in 3.7s
attempt 2: retry in 5.1s
attempt 3: retry in 12.9s
attempt 4: retry in 24.3s
-> dead-letter queue (a human should look at this)