Design: a chat system

The server needs to speak first

Every system so far shared one assumption: the client asks, the server answers. Chat breaks it. A message arrives for you, and nobody asked for it - the server has to speak first.

You could poll: ask 'anything new?' every two seconds. It works, it's simple, and it's wasteful - at a million idle users that's 500,000 pointless requests a second, and messages still land up to two seconds late.

A WebSocket is a connection that stays open, either side able to send at any time. One connection per user, mostly idle, and a message is pushed the instant it exists. The cost is that connections are now state: your load balancer can no longer treat servers as interchangeable, because a user's socket lives on one specific machine.

Imagine it like this: Polling is ringing the post office every two minutes to ask if anything came. A WebSocket is leaving the line open so they can just tell you.

Words you just learned

polling: - the client repeatedly asking whether anything is new
WebSocket: - a long-lived two-way connection where either side may send
presence: - tracking who is currently online and where their connection lives

Finding the machine that holds the socket

Ana sends Sam a message. Ana's socket is on server 3; Sam's is on server 47. Server 3 has no idea Sam exists. This is the routing problem, and it is the core of chat at scale.

The answer is a presence registry: a fast shared store mapping user -> server. Sam connects to server 47, which writes user:7 -> server-47. Server 3 looks it up and forwards the message to server 47, which pushes it down Sam's open socket.

The entries must expire on their own. If server 47 dies, nothing runs to clean up - so presence carries a short TTL and each server refreshes its own users. A crashed server's entries simply age out, and its users reconnect elsewhere. This is the heartbeat pattern, the same one the extension uses to tell flamee.in it's alive.

routing.py

python

PRESENCE_TTL = 30   # seconds; refreshed by a heartbeat

def on_connect(user_id, server_id):
    presence.set(f"user:{user_id}", server_id, ttl=PRESENCE_TTL)

def send(from_user, to_user, text):
    msg = db.insert_message(from_user, to_user, text)   # persist FIRST

    server = presence.get(f"user:{to_user}")
    if server is None:
        push_notification(to_user, text)                # offline
    else:
        forward_to(server, msg)                         # they're online
    return msg.id

What it prints

recipient online   -> forwarded to server-47, delivered in ~40 ms
recipient offline  -> stored + push notification
server crashed     -> presence key expires in 30s, user reconnects

The message is persisted before it is routed. If delivery fails, the message still exists and is fetched on reconnect. Never let a network hop be the only copy of someone's message.

Words you just learned

presence registry: - a fast store mapping each online user to the server holding their socket
heartbeat: - a periodic 'still here' that keeps a presence entry from expiring

Ordering, and the ticks users can feel

Messages must appear in the order they were sent, and clock time cannot give you that. Two servers' clocks differ by milliseconds, so sorting by created_at will eventually show a reply above the message it answers.

Order per conversation, not globally. A global order across all chats is enormously expensive and nobody can perceive it. Within one conversation, a per-conversation sequence number gives a total order that is cheap and exactly as correct as users can tell.

Then there are the ticks - and they are not decoration, they are the delivery guarantees made visible. Sent means the server has it. Delivered means the recipient's device acknowledged it. Read means their client says it was displayed. Each tick is a real acknowledgement travelling back, which is why they can stall independently and why a message can sit on one tick while the recipient is offline.

ordering.sql

sql

-- Per-conversation sequence: cheap, and all a user can perceive
CREATE TABLE messages (
  conversation_id BIGINT NOT NULL,
  seq             BIGINT NOT NULL,   -- 1, 2, 3... within THIS conversation
  sender_id       BIGINT NOT NULL,
  body            TEXT   NOT NULL,
  created_at      TIMESTAMPTZ DEFAULT now(),
  PRIMARY KEY (conversation_id, seq)
);

-- Reconnecting after a dropped socket: "what did I miss?"
SELECT seq, sender_id, body FROM messages
WHERE conversation_id = 42 AND seq > 118
ORDER BY seq;

What it prints

seq | sender_id | body
----+-----------+------------------------
119 |         7 | did you see the deploy?
120 |         3 | yes - it's green
121 |         7 | 🎉

The primary key (conversation_id, seq) shards naturally by conversation and makes 'what did I miss since 118' a single indexed range read. Reconnection stops being a special case and becomes an ordinary query.

The whole course, in one system

Chat uses every piece: a load balancer with sticky routing (module 1), a database sharded by conversation (module 2), a queue for push notifications (module 3), and idempotency keys so a resent message isn't stored twice. Nothing here was new - it was arrangement.

Remember these

WebSockets over polling once connections outnumber messages
A presence registry with TTLs routes messages between servers
Persist the message before routing it - never trust one hop
Order per conversation; global ordering is expensive and imperceptible
Sent / delivered / read are acknowledgements, not UI decoration