API Idempotency Keys: Prevent Duplicate Orders and Payments

A client sends an order request, the server completes it, but the response is lost in transit. The client sees only a timeout and retries. If the API creates another order or charges the card twice, the fault lies in server design, not in the client's reasonable retry behavior.

An Idempotency Key lets multiple attempts of one operation produce one business effect. The server recognizes a retry using a client-supplied key, compares the request, and returns the stored result instead of executing again.

What is idempotency?

An idempotent operation has the same intended effect after one or many identical executions. HTTP defines GET, PUT, and DELETE as idempotent in semantics, while POST commonly is not. An idempotent method does not make an implementation automatically safe: a DELETE endpoint that emails on every call still repeats a side effect.

Idempotency Keys are especially useful when creating orders, bookings, invoices, payouts, charges, refunds, infrastructure resources, and any POST/PATCH request that a client, proxy, or queue may retry.

The goal is not to block every duplicate request; it is to ensure retries of the same intent do not create additional effects.

1. The client–server contract

The client generates a high-entropy random key, commonly a UUID, and keeps it unchanged across retries:

POST /v1/orders HTTP/1.1
Authorization: Bearer ...
Idempotency-Key: 8e03978e-40d5-43e8-bc93-6894a57f9324
Content-Type: application/json

{
  "cart_id": "cart-9821",
  "shipping_address_id": "address-17"
}

A new business operation needs a new key. Avoid short timestamps, counters, or payload hashes as keys: collisions and predictability are risks, and two valid intents may have identical payloads.

The server must document format, maximum length, supported operations, and retention. The IETF has discussed Idempotency-Key in an Internet-Draft; it remains work in progress rather than a completed RFC.

2. Scope keys by tenant and operation

Do not use the client key as a globally unique lookup by itself. Different customers can generate the same UUID, and the same key on separate endpoints may represent separate intents. Use a composite identity:

(tenant_id, operation, idempotency_key)

Derive tenant_id from the authenticated principal, not an arbitrary request field. Use a stable operation name such as orders.create rather than a dynamic URL.

3. Store an idempotency record

CREATE TABLE idempotency_keys (
    tenant_id uuid NOT NULL,
    operation varchar(100) NOT NULL,
    idempotency_key varchar(255) NOT NULL,
    request_hash char(64) NOT NULL,
    status varchar(20) NOT NULL,
    resource_type varchar(100),
    resource_id varchar(100),
    response_code integer,
    response_headers jsonb,
    response_body jsonb,
    locked_until timestamptz,
    created_at timestamptz NOT NULL DEFAULT now(),
    completed_at timestamptz,
    expires_at timestamptz NOT NULL,
    PRIMARY KEY (tenant_id, operation, idempotency_key)
);

CREATE INDEX idx_idempotency_expiry
ON idempotency_keys (expires_at);

Common states are processing, completed, and optionally failed. The request hash prevents reuse with a different payload. Store either the resource ID for reconstruction or the status and body for exact replay.

4. Create a stable request fingerprint

Hashing raw JSON is unstable because field order and whitespace can differ without changing meaning. Parse and validate JSON, normalize allowed fields and defaults, sort object keys deterministically, include the method/operation and meaningful parameters, then hash the canonical representation with SHA-256:

fingerprint = SHA256(
    operation + "\n" + canonical_json(validated_input)
)

Do not hash authorization tokens or irrelevant volatile headers. For large uploads, use a verified content checksum and metadata. If a key matches but the fingerprint differs, return a conflict; never replay the first response for the second payload.

5. Claim the key with a uniqueness constraint

Two identical requests may arrive concurrently. Application-level check-then-insert races; let the database constraint arbitrate:

INSERT INTO idempotency_keys (
    tenant_id, operation, idempotency_key,
    request_hash, status, locked_until, expires_at
) VALUES (
    :tenant_id, :operation, :key,
    :hash, 'processing', now() + interval '30 seconds',
    now() + interval '24 hours'
)
ON CONFLICT DO NOTHING;

The successful insert owns execution. A losing request reads the existing row. A different hash receives 409 Conflict; completed returns the stored result; processing either waits briefly or returns an in-progress conflict. Take over an expired lease only when the previous effect can be determined safely.

6. Keep business data in the transaction

For a single-database operation, create the business record and complete the idempotency row atomically:

BEGIN;

SELECT * FROM idempotency_keys
WHERE tenant_id = :tenant
  AND operation = 'orders.create'
  AND idempotency_key = :key
FOR UPDATE;

INSERT INTO orders (...) VALUES (...)
RETURNING id;

UPDATE idempotency_keys
SET status = 'completed',
    resource_type = 'order',
    resource_id = :order_id,
    response_code = 201,
    response_body = :body,
    completed_at = now()
WHERE tenant_id = :tenant
  AND operation = 'orders.create'
  AND idempotency_key = :key;

COMMIT;

A crash before commit rolls back both records. A crash after commit but before response delivery leaves a completed row that can replay the created order.

7. Full response or resource reference?

Full response storage replays the exact status and body but consumes storage and may retain personal data. Resource references are smaller, yet reconstructed responses may differ after the resource changes. A hybrid stores the ID, status, and minimum response snapshot required by the contract.

Never replay sensitive or short-lived headers such as Set-Cookie. Allowlist safe headers instead of serializing everything.

8. Should errors be cached?

• Validation before execution: usually do not claim or persist the key.
• Final business rejection: storing a 4xx decision can produce consistent retries.
• 5xx before any effect: execution can generally be retried.
• Failure after an external effect: persist enough state to reconcile rather than blindly repeat.

Some APIs, including Stripe, store status and body once endpoint execution begins, including 500 responses. Do not copy a policy mechanically; base it on when your system can prove whether effects occurred and document it for clients.

9. Integrate payment providers

Your API's key and the provider's key protect different boundaries. Derive a stable provider key from an internal payment attempt rather than creating a new UUID on each worker retry:

provider_key = "payment-attempt:" + payment_attempt_id

Persist provider request IDs and state. When a timeout leaves charge status unknown, query the provider by key or request ID before issuing another command. Deduplicate provider webhooks by their event IDs.

Do not hold a database transaction open during a payment network call. Commit intent through a state machine and transactional outbox, then let a worker perform the idempotent external effect.

10. Idempotency and Transactional Outbox complement each other

Idempotency protects inbound API retries. Outbox reliably publishes events after commit. One order transaction can claim the key, create the order, insert OrderCreated, store the idempotent response, and commit everything together.

Event consumers still require their own deduplication because brokers may redeliver. One idempotency key does not automatically propagate guarantees across every service.

11. TTL and key reuse

Retention must exceed the actual retry window for clients, queues, offline mobile apps, and webhooks. Twenty-four hours may fit interactive APIs; payouts or long provisioning may need more. Publish the retention policy.

After deletion, reusing a key can create a new operation, so clients should never recycle keys. High-value transactions should also have a long-lived unique business identifier such as merchant_order_id.

DELETE FROM idempotency_keys
WHERE expires_at < now()
  AND status <> 'processing'
LIMIT 5000;

Clean in batches or partitions and never delete unreconciled processing rows.

12. Security and abuse

• Limit key length and format to prevent storage abuse.
• Scope lookups by authenticated principal.
• Recheck current authorization before returning a cached response.
• Rate-limit new keys because attackers can send endless UUIDs.
• Minimize or encrypt personal data in stored responses.
• Do not log payment bodies merely for idempotency debugging.

A key is not a credential. Knowing it must never grant access to a resource.

13. Observability

• New, replayed, conflicting, and in-progress request ratios.
• Processing-to-completed latency.
• Expired leases and operations needing reconciliation.
• Table size, insertion/deletion rate, and record age.
• Duplicates prevented by operation, not raw key.
• Provider timeouts and lookup outcomes.

Keep request IDs and trace IDs separate. A request ID identifies one attempt; an idempotency key groups attempts of one logical operation.

14. Required test scenarios

1. Two concurrent identical requests create one resource.
2. The same key with a different payload returns conflict.
3. A crash before commit leaves no business record.
4. A crash after commit but before response replays correctly.
5. A payment-provider timeout reconciles without a second charge.
6. A retry after TTL follows the documented policy.
7. Authorization changes do not expose an old response.
8. Cleanup never removes a processing request.

Production checklist

• Clients generate high-entropy keys and preserve them across retries.
• The server scopes by tenant, operation, and key.
• Canonical payload fingerprints are compared.
• A uniqueness constraint handles races.
• Business data and idempotency results commit atomically where possible.
• 4xx, 5xx, in-progress, and lease-expiry policies are explicit.
• External effects have separate provider keys and state machines.
• TTL, cleanup, security, and metrics are operationalized.

Conclusion

Idempotency Keys turn retries from a risk into a normal part of fault-tolerant APIs. Correct design requires more than a cache: scope the key, fingerprint the request, resolve races with uniqueness constraints, connect business effects to transactions, and model external side effects separately. Idempotency does not promise that code executes exactly once; it ensures multiple attempts of one intent produce one business outcome.

References

• Stripe: Idempotent requests
• IETF Internet-Draft: Idempotency-Key HTTP Header Field
• RFC 9110: HTTP Semantics