Modern businesses run on connected systems: the store talks to the ERP, the CRM talks to the billing platform, the warehouse reports back to all of them. When an integration fails loudly, someone fixes it. The expensive failures are the quiet ones: an order created twice, a payment that never updated the invoice, stock levels that drift apart over weeks.
Almost all of them come down to a handful of patterns being skipped.
Assume every message can arrive twice
Networks fail between sending a request and receiving the reply. The sender cannot tell whether the request succeeded, so it retries. Webhook providers deliver at least once, which means sometimes more than once. Your integration must be idempotent: processing the same message twice must have the same effect as processing it once.
- Record the unique ID of every event or request you process, and skip ones you have already handled.
- When calling APIs that support it, send an idempotency key with requests that create things, such as payments or orders, so a retry cannot create a duplicate. Stripe is a well-known example of an API that supports this.
- Use natural unique constraints in your database, such as one invoice per order number, as a final safety net.
Handle webhooks properly
- Verify the signature on every webhook using the provider’s shared secret, and reject old timestamps to prevent replays.
- Respond quickly with a success code, then do the real work in a background job. Slow responses cause timeouts and retries.
- Do not rely on order: events can arrive out of sequence. Compare timestamps or versions, or fetch the latest state from the API before acting.
- Run a periodic reconciliation job that compares both systems, because some webhooks will be missed.
Retry with backoff, and know when to stop
- Retry temporary failures, such as timeouts and 5xx errors, with exponential backoff and random jitter, so many clients do not retry at the same moment.
- Respect rate limits: on a 429 Too Many Requests response, wait for the period given in the Retry-After header when one is provided.
- Do not retry errors that will never succeed, such as validation failures; send them to a person.
- Move messages that keep failing to a dead-letter queue with an alert, instead of retrying forever.
Make integrations observable
Log every message with a correlation ID that follows it across systems, so one order can be traced end to end. Monitor queue depth, failure counts and the age of the oldest unprocessed message. A dashboard that shows the integration is healthy is worth more than any amount of retry logic, because it tells you when something is wrong before your customers do.
