Idempotency
The property of an operation whereby executing it multiple times produces the same result as executing it once — a critical design requirement for resilient distributed systems where retries and duplicate deliveries are inevitable.
Problem
In distributed systems, “did the request succeed?” is often ambiguous:
- A network failure can occur after the server processes a request but before the client receives the response.
- Message brokers guarantee at-least-once delivery, meaning duplicates can arrive.
- Retries are essential for resilience, but retrying a non-idempotent operation (like “charge $100”) can cause double-charges or duplicate records.
Solution / Explanation
Design every operation that must be retryable to be idempotent: the first call and every subsequent call with the same input produces the same outcome.
Idempotent vs. Non-Idempotent Operations
| Operation | Idempotent? | Reason |
|---|---|---|
GET /orders/123 | Yes | Read operations are naturally idempotent |
PUT /orders/123 {status: "shipped"} | Yes | Setting a value to a specific state is idempotent |
DELETE /orders/123 | Yes (HTTP spec) | Deleting an already-deleted resource = 404, same effect |
POST /orders (create) | No (by default) | Creates a new record each time |
POST /payments/charge | No (by default) | Charges the card each time |
Idempotency Keys
For non-idempotent operations, clients send a unique idempotency key (UUID or similar) with the request. The server:
- Checks if this key has been processed before.
- If yes, returns the original response (cached).
- If no, processes the request, stores the result against the key, and returns the response.
This makes any operation idempotent from the client’s perspective.
Idempotent Consumer (Message Processing)
When consuming messages from a broker, the consumer must handle duplicates. Strategies:
- Outbox / Inbox with deduplication: Inbox Pattern stores message IDs and rejects re-processing of seen IDs.
- Natural idempotency: design the handler so processing the same event twice causes no harm (e.g., “set status to X” rather than “increment counter”).
- Conditional writes: only apply the update if the current state matches the expected pre-condition.
Exactly-Once vs. At-Least-Once
Message brokers commonly guarantee at-least-once delivery (a message may be delivered more than once). Exactly-once semantics require distributed transactions or idempotent consumers — the latter is almost always preferred.
Key Components
- Idempotency key — a unique identifier per logical operation, generated by the client.
- Idempotency store — server-side storage (database, cache) mapping keys to results.
- TTL — keys expire after a window; old duplicates beyond the window are not guaranteed.
- Idempotent consumer — a message handler that is safe to run multiple times with the same message.
When to Use
- Payment processing, order creation — any operation with financial or legal consequences.
- Message consumers in event-driven systems (at-least-once brokers = always need idempotent consumers).
- API endpoints that will be called from mobile clients over unreliable networks.
- Any operation with retry logic.
Trade-offs
| Benefit | Drawback |
|---|---|
| Safe to retry without side effects | Requires idempotency key storage and lookup |
| Works correctly with at-least-once delivery | Keys must be managed and expired appropriately |
| Foundation for saga compensating transactions | Natural idempotency is not always achievable |
| Enables reliable distributed workflows | Concurrency control for the first write requires care |
Related
- Outbox Pattern — publishes events exactly once via the transactional outbox
- Inbox Pattern — deduplicates incoming messages on the consumer side
- Saga Pattern — saga steps must be idempotent to handle retries
- Eventual Consistency — at-least-once delivery and idempotency go together
- Resiliency Patterns — idempotency is a foundational resiliency property