A timeout does not mean that an operation failed. It means the caller stopped waiting without learning the operation’s outcome.

That distinction is the source of a common distributed-systems failure. A client sends POST /payments, the server commits the payment, and the response disappears with a dropped connection. If the client retries as a new operation, the system can create a second payment. If it never retries, it may report failure even though money moved.

Retry safety is therefore not a loop around an HTTP call. It is a protocol between the caller, the API, storage, and any downstream message consumers. The protocol needs a stable operation identity, atomic deduplication, a defined replay policy, and bounded load when failure affects many clients at once.

HTTP method semantics are the starting point, not the whole design

RFC 9110 defines a method as idempotent when multiple identical requests have the same intended effect on the server as one request. PUT, DELETE, and safe methods such as GET are idempotent by their standardized semantics. The server may still log every request, and the responses to repeated requests do not have to be byte-for-byte identical.

This is different from saying:

  • every endpoint using PUT is implemented correctly;
  • every POST is unsafe to retry;
  • an idempotent request executes only once;
  • the caller will receive the same response on every attempt.

DELETE /documents/42 can return 204 on the first attempt and 404 on a later attempt while retaining the same intended effect: document 42 is absent. Conversely, a badly designed GET /jobs/run-cleanup can have a non-idempotent effect despite using a method that is supposed to be safe.

For operations such as “create a payment,” “submit an order,” or “start a deployment,” the method is usually POST because the server assigns a resource or initiates a command. Retry safety then needs an application-level contract. An idempotency key is the caller’s declaration that several HTTP requests are attempts to perform one logical operation.

Work the ambiguous timeout by hand

Consider a client creating order ord_8421 with key k_7f91.

Timeline A: the write commits and the response is lost

12:00:00.000  Client -> API: POST /orders
              Idempotency-Key: k_7f91

12:00:00.030  API inserts deduplication record for k_7f91
12:00:00.045  API inserts order ord_8421
12:00:00.052  Database transaction commits
12:00:00.058  API -> Client: 201 Created { "id": "ord_8421" }
12:00:01.000  Client read deadline expires; response was lost
12:00:01.300  Client retries with the same key k_7f91
12:00:01.315  API finds the completed deduplication record
12:00:01.318  API -> Client: replayed 201 { "id": "ord_8421" }

The second HTTP request is not a second operation. It is another attempt to learn the result of the operation identified by k_7f91.

Timeline B: the first transaction rolls back

12:00:00.000  Client -> API: POST /orders, key k_7f91
12:00:00.030  API starts a database transaction
12:00:00.041  Process crashes before commit
12:00:00.042  Database rolls back the open transaction
12:00:01.000  Client times out
12:00:01.300  Client retries with key k_7f91
12:00:01.315  No committed deduplication record exists
12:00:01.350  API creates the order and commits
12:00:01.360  API -> Client: 201 Created

The same client behavior is correct in both timelines because the server couples the deduplication record and business mutation in one atomic transaction. Without that coupling, there are two fatal gaps:

  1. Record the key, crash, and never create the order. Every retry is suppressed.
  2. Create the order, crash, and never record the key. A retry creates another order.

The key identifies intent, not request similarity

A server could hash request fields and assume matching payloads are duplicates. That fails when two legitimate operations happen to have the same parameters. A user may intentionally buy two identical tickets or transfer the same amount twice.

The caller is the only participant that knows whether two attempts express the same intent. It should generate a high-entropy key before the first attempt, persist it with its local operation state, and reuse it for every retry. A new intended operation receives a new key.

The server should scope the key. A practical uniqueness boundary is:

(tenant_id, operation_name, idempotency_key)

Scoping prevents one tenant from colliding with another and allows the same random value to be used on unrelated operations. operation_name should be a stable semantic identifier such as orders.create.v1, not an unnormalized raw URL.

The server must also bind the key to the original request. Reusing k_7f91 with a different amount or recipient is not a retry; it is a client bug or misuse. AWS describes storing the original parameters and rejecting a later parameter mismatch. Stripe’s official API documentation describes the same check.

Hash a canonical semantic representation rather than raw request bytes:

fingerprint = SHA-256(
  canonical_json({
    "operation": "orders.create.v1",
    "customer_id": "cus_19",
    "currency": "USD",
    "line_items": [
      {"sku": "book-1", "quantity": 1}
    ]
  })
)

Canonicalization must define object-key order, number representation, omitted defaults, and which headers affect semantics. Exclude transport-only fields such as trace IDs. Include API version or operation version so a key cannot silently cross incompatible semantics.

Do not put email addresses, account numbers, or other sensitive data in the key. Treat it as an opaque identifier that will appear in logs and indexes.

A deduplication record is part of the API’s state

For a relational implementation, a compact schema might be:

CREATE TABLE api_idempotency (
    tenant_id       UUID        NOT NULL,
    operation_name  TEXT        NOT NULL,
    idempotency_key TEXT        NOT NULL,
    request_hash    BYTEA       NOT NULL,
    state           TEXT        NOT NULL
                    CHECK (state IN ('in_progress', 'succeeded', 'failed_final')),
    resource_id     TEXT,
    http_status     SMALLINT,
    response_body   JSONB,
    created_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
    expires_at      TIMESTAMPTZ NOT NULL,
    PRIMARY KEY (tenant_id, operation_name, idempotency_key)
);

CREATE INDEX api_idempotency_expiry
    ON api_idempotency (expires_at);

The primary key arbitrates concurrent first attempts. The request hash detects a changed payload. The stored status and response define replay behavior. A resource identifier supports later reconciliation even if retaining the full response is undesirable.

Storage policy is part of the public contract:

  • How long may clients retry with the same key?
  • Does the API replay successful responses only, or terminal failures too?
  • What happens while the original request is still running?
  • What status represents a key reused with different parameters?
  • Can a response be reconstructed after its snapshot is deleted?

Stripe documents one concrete policy for its API v1: it stores the first status code and body after endpoint execution starts, including 500 responses, compares later parameters, and allows pruning after a documented retention interval. That is an example, not a universal rule. An internal API may choose a different policy, but clients need to know it.

Retention must cover the maximum retry and reconciliation window. After a record expires, the same key can become a new operation. If a delayed retry may arrive after seven days, a 24-hour deduplication window is not sufficient merely because another API uses 24 hours.

Make deduplication and the local mutation atomic

For a fast operation whose authoritative state is in one database, the handler can use a single transaction:

function createOrder(tenant, key, request):
    validateRequest(request)  // invalid input has not started the operation
    hash = fingerprint("orders.create.v1", request)

    begin transaction

    inserted = insert api_idempotency(
        tenant, "orders.create.v1", key, hash, "in_progress"
    ) on conflict do nothing

    if not inserted:
        prior = select api_idempotency
                where tenant_id = :tenant_id
                  and operation_name = "orders.create.v1"
                  and idempotency_key = :key

        if prior.request_hash != hash:
            rollback
            return KEY_REUSED_WITH_DIFFERENT_REQUEST

        if prior.state == "succeeded" or prior.state == "failed_final":
            rollback
            return prior.http_status, prior.response_body

        rollback
        return OPERATION_IN_PROGRESS

    order = insert orders(...)
    event = insert outbox(event_id, aggregate_id, type, payload)

    response = {"id": order.id, "status": order.status}
    update api_idempotency
       set state = "succeeded",
           resource_id = order.id,
           http_status = 201,
           response_body = response
     where tenant_id = tenant
       and operation_name = "orders.create.v1"
       and idempotency_key = key

    commit
    return 201, response

The unique insert may block behind a concurrent transaction using the same key. If the first transaction commits, the second observes the conflict and replays its result. If the first rolls back, the second can acquire the key and execute. Exact behavior depends on the database and isolation level, so test the race on the actual engine rather than relying on ORM-level “find then insert” logic.

Do not perform an unprotected check followed by an insert:

if not exists(key):       // request A: absent; request B: absent
    perform_side_effect() // both execute
    insert(key)

The database uniqueness constraint—not a process-local lock—is the final concurrency guard across application instances.

Long-running operations need a different shape. Commit an idempotency record and a durable job in one transaction, return 202 Accepted, and let repeated requests retrieve the same job identity. Do not hold a database transaction open for minutes while waiting on another service.

An idempotency table cannot atomically cover a remote side effect

The transaction above works because the order, idempotency record, and outbox row share a database. It does not make “write PostgreSQL and publish to a broker” atomic. This common sequence still loses data:

1. COMMIT order
2. publish OrderCreated

If the process crashes between steps 1 and 2, the order exists but downstream systems never learn about it. Reversing the steps creates the opposite problem: the message may be consumed even if the order transaction rolls back.

The transactional outbox pattern moves the event into the same transaction as the business mutation. A relay publishes committed outbox rows later.

CREATE TABLE outbox (
    event_id       UUID        PRIMARY KEY,
    aggregate_type TEXT        NOT NULL,
    aggregate_id   TEXT        NOT NULL,
    aggregate_seq  BIGINT      NOT NULL,
    event_type     TEXT        NOT NULL,
    payload        JSONB       NOT NULL,
    created_at     TIMESTAMPTZ NOT NULL DEFAULT now(),
    published_at   TIMESTAMPTZ,
    UNIQUE (aggregate_type, aggregate_id, aggregate_seq)
);

event_id is the downstream deduplication identity. aggregate_seq makes ordering requirements explicit per aggregate. A polling publisher can claim batches, while change data capture can derive the event stream from committed database-log entries. Debezium’s official outbox documentation describes the latter implementation.

The outbox solves the atomic dual-write problem, but it does not guarantee that the broker sees an event only once:

13:00:00.000  Relay publishes event e_44
13:00:00.020  Broker acknowledges e_44
13:00:00.021  Relay crashes before setting published_at
13:00:05.000  Replacement relay sees e_44 as unpublished
13:00:05.020  Relay publishes e_44 again

Chris Richardson’s transactional outbox pattern explicitly identifies this crash window. The consumer must therefore be idempotent. An inbox or processed-message table can couple deduplication to the consumer’s local mutation:

CREATE TABLE processed_messages (
    consumer_name TEXT        NOT NULL,
    event_id      UUID        NOT NULL,
    processed_at  TIMESTAMPTZ NOT NULL DEFAULT now(),
    PRIMARY KEY (consumer_name, event_id)
);
function handleOrderCreated(event):
    begin transaction

    inserted = insert processed_messages("billing", event.id)
               on conflict do nothing

    if not inserted:
        rollback
        acknowledge event
        return

    applyLocalBillingState(event)
    commit
    acknowledge event

If the consumer itself must call another non-transactional service, the problem repeats at that boundary. Use the downstream service’s idempotency contract or write another outbox. “Exactly once” is not inherited across arbitrary side effects just because one broker offers an exactly-once feature.

Retry only where another attempt can help

Idempotency makes retries safe from duplicate intended effects; it does not make retries free. During overload, immediate retries consume additional connections, CPU, and queue capacity. If every layer retries independently, attempts multiply. Three total attempts at each of five layers can produce up to 3⁵ = 243 calls at the deepest dependency for one top-level operation.

A retry policy needs:

  • an end-to-end deadline;
  • a per-attempt timeout that leaves time for another attempt;
  • a small maximum attempt count;
  • an explicit set of retryable failures;
  • exponential backoff capped at a maximum delay;
  • jitter to prevent synchronized clients from retrying together;
  • one chosen retry layer for a given dependency path.

A common full-jitter calculation is:

cap(attempt) = min(max_delay, base_delay * 2^attempt)
sleep        = random_uniform(0, cap(attempt))

AWS’s Builders’ Library explains why backoff reduces pressure on a failing dependency and why jitter spreads clustered retries. The cap prevents delays from growing without bound. The attempt limit and overall deadline prevent a client from retrying forever.

Consider base_delay = 100 ms, max_delay = 2 s, and three retries:

attempt 0 fails at t=0 ms
  cap = 100 ms; sampled sleep = 63 ms

attempt 1 fails at t=143 ms after an 80 ms call
  cap = 200 ms; sampled sleep = 27 ms

attempt 2 fails at t=250 ms after an 80 ms call
  cap = 400 ms; sampled sleep = 311 ms

attempt 3 starts at t=561 ms

Another client will probably sample different sleeps. Deterministic exponential backoff would still align clients that failed at the same time.

Retry ambiguous response timeouts, throttling responses, and selected server errors only when the operation is idempotent and the deadline permits. A connection failure is safe for a non-idempotent operation only when the client can establish that the original request was never applied; RFC 9110 warns against guessing. Honor a valid Retry-After when the server provides one. Most validation and authorization failures will not improve with another identical attempt. A 500 is not automatically transient, and an API that stores the first response for an idempotency key may replay that same failure. The API’s documented contract takes priority over a generic status-code list.

The client also needs durable operation state

Generating a random key inside each HTTP attempt defeats the protocol:

retry(() => post("/orders", key=randomUUID()))  // wrong

Generate once outside the loop:

key = persistedOperation.idempotencyKey

for attempt in 0..maxRetries:
    remaining = deadline - now()
    if remaining <= minimumAttemptBudget:
        return OUTCOME_UNKNOWN

    result = post("/orders", key, body, timeout=attemptTimeout(remaining))

    if result.isDefinitive:
        persistOutcome(result)
        return result

    if not result.isRetryable:
        return result

    sleep(fullJitter(attempt), boundedBy=remaining)

return OUTCOME_UNKNOWN

OUTCOME_UNKNOWN is a real state, not an error message to hide. A caller that exhausts its deadline should retain the key and reconcile later through another retry or a lookup endpoint. A user interface should not invite the user to “try again” by creating a fresh key when the original result remains ambiguous.

Operational checks make the contract testable

Track facts that reveal whether the design works:

  • first attempts, replays, parameter mismatches, and in-progress collisions;
  • deduplication record age and cleanup volume;
  • retry attempts by operation, dependency, and reason;
  • end-to-end success after retry, not merely per-attempt error rates;
  • outbox age, unpublished row count, relay attempts, and publish latency;
  • duplicate events detected by each consumer;
  • operations left in an unknown or in-progress state beyond their expected duration.

Avoid using raw idempotency keys as unbounded metric labels. Put a hashed or redacted form in structured logs and traces when correlation is necessary.

Test the failure boundaries deliberately:

  1. Crash before the database commit.
  2. Commit, then drop the HTTP response.
  3. Send two concurrent requests with the same key and body.
  4. Reuse one key with a different body.
  5. Publish an outbox event, then crash before marking it published.
  6. Apply a consumer mutation, then crash before acknowledging the message.
  7. Expire a deduplication record, then deliver a late retry.
  8. Fail a dependency for many clients and verify that jitter spreads retry load.

Retry-safe APIs do not eliminate uncertainty from the network. They give every layer enough durable identity and state to resolve that uncertainty without duplicating the intended work.

References