Idempotency & exactly-once (the lie)

Exactly-once delivery does not exist

The moment you added retries, you created a new problem. A worker charges a card, then crashes before it can mark the job done. The queue, quite correctly, has no idea the work happened - so it hands the job to another worker. The customer is charged twice.

The instinct is to want 'exactly-once delivery'. It is not available. Any system that can lose a message can retry it, and no acknowledgement can be both sent and guaranteed to arrive. You get to choose between at-most-once (never duplicated, sometimes lost) and at-least-once (never lost, sometimes duplicated).

Nobody chooses to silently lose payments. So you take at-least-once and make duplicates harmless. That property - doing it twice has the same effect as doing it once - is idempotency, and it is the single most important word in this module.

The idempotency key

The mechanism is simple. The caller invents a unique id for its intent - not for the attempt, for the intent - and sends it with every retry of that same request. The server records which keys it has already completed, and if it sees one twice it returns the original result instead of doing the work again.

The subtlety that catches people: the check and the work must be one atomic step. If you check 'have I seen this key?' and then insert afterwards, two retries arriving together can both pass the check before either inserts. The database's own uniqueness guarantee is what makes this safe - a UNIQUE constraint on the key, and let the second one lose.

CREATE TABLE charges (
  idempotency_key TEXT PRIMARY KEY,   -- the DB enforces "only once"
  user_id         BIGINT NOT NULL,
  amount_cents    INT    NOT NULL,
  created_at      TIMESTAMPTZ DEFAULT now()
);

-- Every retry sends the SAME key. The second one changes nothing.
INSERT INTO charges (idempotency_key, user_id, amount_cents)
VALUES ('order-8821-charge', 7, 4999)
ON CONFLICT (idempotency_key) DO NOTHING
RETURNING id;

first attempt   INSERT 0 1     id = 3310   -> charge the card
retry (crash)   INSERT 0 0     (no rows)   -> already done, return 3310

Some operations are already safe

Not everything needs a key. Look at the shape of the operation first, because some are naturally idempotent and some can be rewritten to be.

SET status = 'shipped' is idempotent - run it a thousand times, the status is shipped. balance = balance - 10 is not - each run takes another ten. That difference is the whole lesson: prefer assignment over adjustment, and absolute values over relative ones.

When you genuinely need an increment, the key gives you back the safety the operation itself lacks. Record that this key was applied, and refuse to apply it a second time.

-- Idempotent: the outcome is the same however many times it runs
UPDATE orders SET status = 'shipped'  WHERE id = 8821;
DELETE FROM carts WHERE user_id = 7;

-- NOT idempotent: every run changes the answer again
UPDATE accounts SET balance = balance - 1000 WHERE id = 7;
INSERT INTO emails (user_id, kind) VALUES (7, 'welcome');

run 3x:  status = 'shipped'        (same)
run 3x:  cart deleted             (same)
run 3x:  balance -3000            (DIFFERENT - charged 3 times)
run 3x:  3 welcome emails sent    (DIFFERENT - user is annoyed)