How to Design Webhook Retries, Ordering, Idempotency [2026]

If you’re here because you searched “design webhook delivery system retries ordering idempotency”, you’re probably not looking for feel-good architecture diagrams. You want a webhook delivery system that can push 10 million events/day without collapsing the first time a customer’s endpoint returns 500 for two hours.

That means sane retries, explicit ordering guarantees (not vibes), and an idempotency contract that still holds when things are on fire. The annoying parts matter most: tenant isolation, retry storms, replay UX, and what you actually measure so you catch incidents before Support does.

The 2026 reality: “send a POST and retry a few times” is table stakes. Svix and Hookdeck basically taught customers to expect Stripe-level delivery timelines, selective replays, and per-endpoint visibility by default. The first time a customer’s webhook goes down and you can’t answer “what happened to my events?”, you’re going to feel it.

What is a webhook delivery system?

A webhook delivery system is the infrastructure that reliably delivers event notifications from your product to customer-owned HTTP endpoints, usually with at-least-once delivery, retries with backoff, and tooling to debug failures and replay events.

At 10M events/day, the average is about 116 events/second. The average is also a lie. Real systems burst. Deploys happen. Imports happen. Billing runs happen. And if you don’t design for bursts, your retry logic turns into a self-inflicted DDoS.

Here’s my stance, and I’m not backing off it: stop chasing “exactly-once webhooks”. You won’t get it across the public internet. What you can get is a clean contract that makes duplicates boring.

• Your platform guarantees at-least-once delivery.
• You scope and publish your ordering guarantees (per endpoint, per tenant, or none).
• You provide a first-class idempotency key (event ID), and receivers are expected to dedupe.

The pipeline: outbox → router → per-tenant scheduling → deliver → observe

When teams tell me they “lost webhooks,” it’s almost always one of two failures:

1) a dual-write bug (DB update succeeded, enqueue failed), or 2) an overload bug (retries ate all worker capacity and fresh events got starved).

So I design this as five explicit stages. No mystery meat “worker does everything” blob.

1) Emit events safely with a transactional outbox

If you write business state in Postgres and then publish the webhook event to a queue “after”, you’ve built a time bomb. Crash at the wrong moment and that event never gets published. You won’t notice until a customer asks why their integration missed an invoice.

Use the transactional outbox pattern. As Chris Richardson lays out, you write the domain event into an outbox table in the same DB transaction as the business update. Then you publish asynchronously.

Concrete design:

• outbox_events table keyed by event_id (UUIDv7 or ULID)
• Columns: tenant_id, event_type, aggregate_id, payload_json, created_at
• An outbox publisher reads in batches of 500–5,000 rows depending on payload size

At 10M/day, if your payload averages 2 KB, that’s ~20 GB/day moving through your system. Don’t shove that much “payload movement” through your OLTP database if you can avoid it. Store minimal payload in the outbox, or store a pointer to object storage for large payloads.

2) Route each event to an ordering domain

Ordering is never global. Anyone promising “ordered webhooks” without specifying scope is selling you a story.

The honest question is: _where does ordering actually matter?_ Common choices:

• Per endpoint ordering: events to the same destination URL are delivered in order
• Per tenant ordering: all events for a tenant are ordered (expensive)
• Per event type ordering: invoices ordered, but analytics.ping can arrive whenever

At 10M/day, per-tenant ordering is how you accidentally cap throughput. Per-endpoint ordering is usually the sweet spot.

This decision drives your queue key. Pick it deliberately.

Kafka partition keys

Kafka ordering is per partition. The Apache docs are explicit: you get ordering within a partition, and scaling is about partitions. Translation: your partition key _defines_ your ordering scope. See the Apache Kafka Documentation for the canonical model.

Practical keys:

• partition_key = endpoint_id for per-endpoint ordering
• partition_key = tenant_id for per-tenant ordering (more contention)

If you need 5,000 deliveries/second at peak, and each partition sustains ~200–1,000 msg/s depending on payload and broker tuning, you’re looking at 10–50 partitions minimum. In practice I’d plan for 100+ so you’re not rewriting everything the moment volume doubles.

SQS FIFO message groups

SQS FIFO gives ordering within a message group ID. That’s nice if you want a managed queue and can live with FIFO throughput constraints. Per AWS, FIFO ordering is within MessageGroupId and deduplication is within a limited time window. Read the AWS SQS FIFO queues documentation.

If you do SQS FIFO, I usually set:

• MessageGroupId = endpoint_id
• MessageDeduplicationId = delivery_id (not payload hash)

The constraint is the feature: if one endpoint is slow, that message group becomes a head-of-line blocker. That’s not a bug. That’s your ordering guarantee doing its job.

Retries: status codes, backoff, jitter, and killing retry storms

Retries are where webhook systems go to die. People treat them like an afterthought. Then they wake up to a retry storm that consumes the whole fleet.

You need two things at the same time:

1) be persistent enough that transient failures don’t lose events, and 2) be polite enough that you don’t amplify your customer’s outage.

What status codes should trigger retries?

My default rules:

• Retry on 408, 429, and 500–599
• Retry on connection errors, DNS failures, TLS failures, and timeouts
• Do not retry on 400, 401, 403, 404, 410, 422. Those are “fix your config” failures

The controversial one is 409. Some APIs use it for concurrency conflicts and want retries. Some use it to mean “duplicate.” Decide per product and document it. Don’t make customers reverse-engineer your intent.

Also, treat 202 Accepted as success if you’re sending fire-and-forget webhooks.

Exponential backoff with full jitter

Backoff without jitter is how you create synchronized retry waves. Retry waves become incidents.

A baseline schedule for a single delivery:

• attempt 1: immediate
• attempt 2: +10s
• attempt 3: +1m
• attempt 4: +5m
• attempt 5: +30m
• attempt 6: +2h

That’s 6 attempts over ~2.6 hours. For payment/billing events, you might want 24 hours with more attempts. For low-value events, cut it down aggressively. Reliability is not “keep hammering forever.”

If you want a managed mental model to copy, Google Cloud Tasks is literally “HTTP delivery with retries and schedules,” including max attempts and backoff. The knobs are described in the Google Cloud Tasks documentation.

Backpressure controls that actually work

At 10M/day, the scary number isn’t 10M. It’s what happens when a small slice of endpoints start failing.

• 2% of 10M = 200,000 events/day entering retry
• If your retry policy averages 4 attempts, that’s 800,000 extra HTTP calls/day
• If the failure is correlated (one big customer’s DNS issue), concurrency spikes hard

Controls I consider mandatory:

1. Max in-flight per endpoint: e.g., 5 concurrent requests per endpoint
2. Max in-flight per tenant: e.g., 50 concurrent across all endpoints
3. Global concurrency ceiling: protects your own egress and NAT gateways
4. Adaptive concurrency on 429/503: cut concurrency by 50% for a cooling window
5. Circuit breaker per endpoint: after N consecutive failures (say 20), stop immediate attempts and push retries out

This is one of those places where the boring answer is actually the right one. You’re building a scheduler, not a “worker pool.”

Ordering vs throughput: choosing keys at 10M/day (Kafka + FIFO examples)

You can’t “guarantee ordering of webhook events” without admitting what you’re willing to sacrifice.

Here’s the tradeoff table I use in design reviews.

Concrete example: if you have 100,000 endpoints and peak 5,000 events/s, per-endpoint ordering with Kafka is totally viable. Hash endpoint_id across maybe 200 partitions and you still preserve per-endpoint ordering because all events for an endpoint map to the same partition.

With SQS FIFO, per-endpoint ordering means each endpoint is effectively its own serial lane. That’s clean, but you need to understand FIFO throughput limits and cost. This is where teams often end up using standard SQS plus their own ordering gates.

Idempotency and deduplication: beyond “use event_id”

Webhook idempotency is the receiver’s ability to safely handle duplicate deliveries (the same logical event arriving multiple times) without double-charging, double-sending emails, or corrupting state.

If you run at-least-once delivery (you will), duplicates are not a rare edge case. They’re normal. Design so duplicates don’t matter.

Dedupe key schema

I like two identifiers:

• event_id: stable for the logical event. Same across retries and replays
• delivery_id: unique per endpoint per attempt chain. Useful for debugging

Suggested shapes:

• event_id = ulid() at emission time
• delivery_id = ulid() when you fan out to endpoints

Your idempotency key for the receiver is event_id (and optionally event_id + endpoint_id if you allow multi-endpoint reuse).

Where to store dedupe state (and for how long)

This is the part most posts skip because it forces you to pick a policy.

Options:

• Redis: fast, cheap per lookup, TTL is built in. Risk: eviction under memory pressure
• Postgres: durable, auditable. Risk: hot index writes at high QPS

At 10M/day, if even 20% of receivers implement dedupe and hit your API for receipts, you can see hundreds of QPS of idempotency lookups. Redis is often the right default.

TTL: set your dedupe TTL to match your maximum replay window.

• Typical: 7 days
• High compliance products: 30–90 days

What if the payload differs for the same event_id?

This happens during bugs and sloppy replays. Pick a stance and document it.

My stance:

• event_id is immutable. If the payload changes, it’s a new event with a new event_id
• If you must re-send with a corrected payload, emit a new event type like invoice.updated.corrected

If you let “same ID, different payload” exist, you make dedupe impossible and you poison customer trust.

Tenant isolation: noisy neighbor protection for failing endpoints

Most webhook writeups talk about retries. They don’t talk about fairness. That’s a mistake.

At scale, one big tenant with a broken endpoint can consume your entire delivery fleet if you let them.

Here’s an isolation model that works well.

Per-tenant buckets + fairness scheduler

• Maintain a per-tenant token bucket for delivery capacity
• Scheduler pulls work with weighted round robin across tenants
• Within a tenant, schedule by endpoint priority (oldest next attempt first is a solid default)

Numbers that are easy to reason about:

• tenant_rps_limit: 50 requests/second
• endpoint_rps_limit: 5 requests/second
• max_in_flight_per_endpoint: 5

If a tenant exceeds their budget, they don’t get dropped. They get delayed. That delay becomes visible in queue_age_seconds.

Quarantine queues for persistent failures

After, say, 100 failed attempts (or 24 hours of failure), stop pretending this is transient. Move deliveries into a quarantine state:

• next attempt every 6–12 hours
• surface in the dashboard as “Action required”

This keeps your hot path clean and your costs predictable.

Replay UX: what customers expect in 2026

If you want Stripe-like delivery UX, you need to store delivery attempts as first-class data. There’s no way around it.

The basic UX objects:

• Events
• Endpoints
• Deliveries (event × endpoint)
• Attempts (delivery × attempt_number)

Data model for deliveries and attempts (debugging + compliance)

A minimal schema:

• webhook_deliveries: delivery_id, event_id, tenant_id, endpoint_id, status (pending/success/failed), next_attempt_at, created_at
• webhook_attempts: attempt_id, delivery_id, attempt_number, request_headers_redacted, request_body_redacted, response_status, response_headers_redacted, response_body_redacted, duration_ms, error_class, created_at

Retention defaults I like:

• Attempts: 7–14 days (payloads are expensive)
• Delivery outcome metadata: 90 days (cheap, useful)

Redaction:

• Always redact Authorization, cookies, and secrets
• Allow tenants to mark JSON paths as sensitive and apply field-level redaction

I use similar patterns in my redaction posts like field-level redaction for RAG pipelines. The same design applies to webhook payload capture.

Safe replay mechanisms

Replays can be an outage amplifier. Make them safe by default:

• Rate limit replays (per tenant and per endpoint)
• Offer a “dry run” that validates signature generation and endpoint connectivity without sending payload
• Bulk replay requires explicit confirmation and shows estimated volume

This is one of those places where product UX doubles as an SRE control. If your replay button can take down an endpoint, you built a footgun.

Security: signatures, timestamps, and replay protection

You’re delivering data to the public internet. Assume it will be intercepted, forwarded, and replayed.

Minimum viable security:

• HMAC signature over (timestamp + raw_body)
• Include a X-Webhook-Timestamp and reject if older than 5 minutes (receiver-side)
• Include event_id so receivers can dedupe

If you want to go further:

• mTLS for high-trust enterprise tenants
• Per-endpoint secret rotation with an overlap window (e.g., accept old secret for 24 hours)

Also, don’t ignore SSRF risks if you provide “test delivery” features. Treat endpoint URLs like untrusted input and apply allowlists and egress controls.

Observability SLIs/SLOs: what to measure so you catch incidents early

If you only have logs, you don’t have a webhook system. You have a support ticket generator.

Here are the SLIs I’d put on a dashboard on day one.

Core SLIs (with concrete thresholds)

1. Delivery success rate (by tenant, by endpoint): target 99.9% over 24h for critical tiers
2. Delivery latency (event created → first successful delivery): p50, p95. A reasonable p95 is < 60s when endpoints are healthy
3. Queue age (oldest pending delivery): alert if > 5 minutes for high tier
4. Retry depth distribution: % deliveries at attempt 1/2/3/4+. Spikes mean widespread failures
5. 429 rate and 5xx rate: broken down by tenant. 429 spikes mean you’re pushing too hard or the receiver is throttling

For SLO management, use error budgets. If you promise 99.9% success, your monthly error budget is ~43 minutes of failure. That’s not a lot. It forces you to treat webhook reliability like a product feature, not an afterthought.

As a nearby pattern, I think the same observability discipline applies to AI in production systems. Reliability is a UX.

What I’d build first (and what I’d postpone)

If you’re not Stripe, don’t build Stripe on day one. But don’t ship a toy either.

My “ship it” MVP for a serious platform:

1. Transactional outbox
2. At-least-once delivery
3. Per-endpoint ordering
4. Retries with exponential backoff + jitter
5. Per-tenant and per-endpoint rate limits
6. Delivery/attempt tables + basic dashboard
7. Selective replay for a single event

Then add the fancier stuff when you’ve earned it:

• Bulk replays
• Enterprise security (mTLS)
• Advanced redaction policies
• Multi-region active-active

I’ve shipped enough internal tooling to know that scaffolding beats policy docs. At Rise People, the compliance scaffolding we baked into project templates got adopted org-wide. The parallel here is straightforward: if you bake reliability into the pipeline (queues, schedulers, caps), you won’t be “fixing webhook reliability” every quarter.

Closing: the challenge

If you’re building a webhook platform in 2026, you’re not competing on “can we POST JSON.” You’re competing on _how well you behave during someone else’s outage_.

So here’s the challenge. Write down, in one sentence, your ordering guarantee and your retry contract. If you can’t say it clearly, you haven’t designed it yet. And your on-call rotation is going to pay for that ambiguity.

Internal links you might find useful while you’re here:

• Reliability thinking for complex systems: AI agents (same failure modes, different domain)
• Observability patterns: [How to Build Vendor-Neutral LLM Observability Monitoring [2026]](/blog/llm-observability-vendor-neutral)
• Redaction strategies: [LLM Data Leakage Playbook [2026]: Logging, Retention, Redaction](/blog/llm-data-leakage-playbook)
• Security regression discipline: prompt injection
• Tooling mindset: Claude Code
• Local-first reliability instincts: local LLM
• Retrieval systems (for replay/search UIs): RAG
• Backpressure on agent workflows: AI agent control flow patterns

Photo by Bernd 📷 Dittrich on Unsplash.