idempotency-and-retry-safety

Idempotency and Retry Safety

Distributed systems cannot cheaply guarantee exactly-once delivery — a client that times out waiting for a response genuinely cannot tell whether the server never received the request or received it and crashed before replying. The practical, achievable guarantee is at-least-once delivery plus idempotent processing on the receiving end, which together behave like exactly-once from the caller's point of view.

At-least-once vs. exactly-once: why this matters

• At-most-once: send once, never retry. Simple, but any dropped request or timeout is a silent failure — unacceptable for anything that charges a card or books a room.
• Exactly-once delivery, as a network-level guarantee, is not achievable in general in an asynchronous network with possible partial failures (the client cannot distinguish "server never got it" from "server got it, processed it, and the ack was lost") — treat any vendor claiming true "exactly-once" as actually meaning "at-least-once delivery with idempotent processing," which is the real, achievable target.
• At-least-once + idempotency is the standard, achievable pattern: the client retries freely on any ambiguous failure (timeout, 5xx, connection reset), and the server is designed so that processing the same logical request N times has the same effect as processing it once.

Designing an idempotency key

1. The key identifies one logical operation, generated by the client, not the server — a UUID created once when the user clicks "book" and attached to every retry of that same booking attempt. If the server generates the key, retries of a request that never reached the server can't reuse it, defeating the purpose.
2. The key's scope must be unambiguous: usually (idempotency_key, endpoint) or (idempotency_key, account_id) — a key that's only unique per-client but collides across unrelated accounts is a data-leak risk, not just a bug.
3. Store the key with the result, not just a "seen it" flag. On a duplicate request, replay the original stored response (same status code, same body) — don't just skip the operation and return a bare 200, which changes behavior from the client's perspective between the first and second attempt.
4. Set a retention window for stored idempotency keys (commonly 24 hours) and reject key reuse with a different request payload within that window as a client error — reusing a key with different parameters usually indicates a client bug, not a legitimate retry.
5. Make the storage write and the business-effect write atomic — in Postgres, insert the idempotency-key row and perform the operation inside the same transaction (or make the idempotency key itself a unique constraint on the effect table) so a crash between the two never leaves the key "claimed" without the effect having happened, or vice versa.