Designing Reliable Webhooks: Signatures, Idempotency, Retries, and Queues

A webhook delivers events over a network, so applications must assume an event can be delayed, duplicated, reordered, or missed on the first attempt. A 200 response does not mean the business operation is complete; a reliable architecture separates ingestion from processing.

A minimal ingestion flow

1. Read the raw request body and required headers.
2. Verify the signature and timestamp tolerance.
3. Check the event ID in an idempotency store.
4. Persist the event and receipt state in a transaction.
5. Enqueue processing.
6. Return a 2xx response quickly.

Stripe notes that modifying the raw body before verification breaks signature checking. Signature algorithms, headers, and tolerances vary by provider, so use the sender's official SDK or documentation.

Idempotency prevents repeated effects

A provider may redeliver an event after a timeout or error. Create a unique constraint on provider + event_id and treat a uniqueness conflict as an already-received event. A separate check followed by insert is vulnerable to races.

BEGIN;
INSERT INTO webhook_events(provider, event_id, payload, status)
VALUES (?, ?, ?, 'received')
ON CONFLICT(provider, event_id) DO NOTHING;
COMMIT;

Idempotency must extend to business effects. Updating an order and marking an event should share a transaction. Email or external calls need a business key or transactional outbox so they are not executed twice.

Respond quickly and process in the background

The endpoint should not create invoices, send email, and synchronize a CRM before responding. Durably persist, enqueue, and return 2xx to avoid timeouts and unnecessary redelivery. A worker can retry with backoff and move permanent failures to a dead-letter queue.

Return 2xx only after the event is durably stored or reliably handed to a queue. A success response before that point can lose the event if the process stops.

Do not assume event order

Events may be processed concurrently or arrive out of sequence. Use versions, timestamps, state machines, and conditional updates to reject stale transitions. When supported, retrieve the resource's current state from the provider API rather than trusting an old snapshot.

Use bounded retries

Retry transient failures such as timeouts, 429, and 5xx responses. Invalid data should be recorded and routed for manual review. Use exponential backoff with jitter, cap the attempt count, and set timeouts on every outbound request.

Secure the endpoint

• Require HTTPS and verify signatures before business parsing.
• Keep secrets in a secret manager and support overlapping rotation.
• Limit body size and subscribed event types.
• Do not log secrets or complete sensitive payloads.
• Treat IP allowlists as an additional layer, not primary authentication.

Observability and operations

Track a correlation ID, received time, attempt count, state, last error, and processing duration. Dashboards should expose verification failures, queue lag, retries, dead letters, and the age of unfinished events. Replay tools need authorization and audit logs and must still pass through idempotency controls.

Required test scenarios

• Bad signatures, expired timestamps, and modified bodies.
• The same event delivered concurrently.
• A worker stopping after a database update but before acknowledgement.
• Out-of-order events.
• Dependencies returning 429, 500, or timeout.
• Replay from the dead-letter queue.

Conclusion

Reliable webhook handling assumes at-least-once delivery: authenticate the sender, persist durably, deduplicate, process asynchronously, and make operations observable. With these layers in place, retries become a recovery mechanism instead of a source of duplicate data.

Reference

• Stripe Docs: Receive events in a webhook endpoint