Webhooks look trivial on paper: accept an HTTP request, run a handler, return 2xx. In production, that mental model breaks fast. You are no longer debugging one request. You are debugging delivery semantics, provider behavior, local network plumbing, and team coordination at the same time.
That gap between "simple API callback" and "distributed event delivery system" is where most teams lose hours. This is the part we rarely document clearly, even though almost every team experiences it.
The illusion of simplicity
The basic setup is easy to explain:
- A provider sends a
POSTrequest. - Your endpoint handles it.
- You persist state and return success.
The tricky part is that providers are explicit about delivery caveats, and those caveats are exactly what local debugging workflows ignore.
- Stripe: retries can continue for up to three days, event order is not guaranteed, and duplicate deliveries can happen (Stripe webhooks docs).
- GitHub: if delivery fails, you need to redeliver manually. GitHub explicitly says failed deliveries are not automatically redelivered (GitHub redelivery docs).
- Local dev: webhook providers need a public URL, so teams usually expose a local listener through a temporary public endpoint.
None of this is exotic. It is documented behavior. The pain comes from the mismatch between provider guarantees and the tooling most teams use day to day.
The real failure modes
Events arrive out of order
Stripe states this directly: event delivery order is not guaranteed. If your logic assumes "created" always arrives before "updated," your local test might pass and production might still fail. This usually appears as impossible states in your database, because your system processed valid events in an invalid sequence.
Retries fire twice, or not at all
Retry behavior differs per provider. Stripe retries automatically for up to three days. GitHub expects manual redelivery when deliveries fail. If your handler is not idempotent, automatic retries duplicate side effects. If your recovery process is manual and inconsistent, you silently miss events. Both failure modes look random until you map provider semantics explicitly.
Local testing becomes tunnel roulette
To test webhooks locally, someone runs a local tunnel command, copies the public URL, and registers it in the provider dashboard. That works for one developer. It gets fragile when a team rotates tunnels frequently or shares one endpoint across multiple machines.
At that point, most teams need a relay-style model: keep one stable public webhook endpoint, then fan events out to each local developer environment.
Nobody knows who caught the event
In many teams, there is no per-developer delivery visibility. An event was sent, but who received it? Was it dropped at the tunnel? Was it rejected by signature verification? Did it hit a stale local process? Without explicit delivery traces per listener, the answer is usually guesswork.
The multi-developer problem
A single listener with multiple developers is a coordination bug disguised as a networking setup.
If three developers are iterating on the same integration and only one active tunnel is registered upstream, two people are effectively debugging stale assumptions. They can run tests locally and still observe nothing, because the event never targeted their environment.
This is the root of the "it works on my machine" webhook variant: the event is real, but delivery ownership is implicit and constantly changing.
The fix is not "better tunnel discipline." The fix is to stop coupling provider configuration to one developer session.
The expensive debugging loop
Most teams eventually converge on the same loop:
- Tail logs.
- Trigger event.
- Realize nothing arrived.
- Restart server.
- Recreate tunnel.
- Re-register webhook URL.
- Retry and hope.
This loop is expensive because each step can fail independently. Most tooling also lacks a single timeline that combines provider delivery, local receipt, handler execution, and retry history.
What good actually looks like
A sane webhook debugging workflow for teams should provide three capabilities by default:
- Broadcast to all active local listeners for a project or environment, so each developer can reproduce the same incoming event stream without owning the single tunnel.
- Full event replay with deterministic payload + headers, so debugging does not depend on waiting for another production trigger.
- Per-developer delivery traces that show where an event failed (ingress, signature verification, handler error, or success), so ownership is clear.
The key design point is explicitness. Delivery should be a visible system, not a hidden side effect of whoever configured the provider dashboard last.
This is the problem space we are building Hookie around: a stable ingress endpoint, fan-out delivery to local listeners, and clear per-developer delivery traces.
Why this matters
Webhook debugging pain is often framed as minor developer-experience friction. It is not minor. It creates missed business events, duplicate side effects, and a lot of false confidence from incomplete local tests.
If your team depends on webhooks for billing, account lifecycle, provisioning, or notifications, this is a reliability problem that shows up first as DX pain. Treating it as infrastructure, not a temporary dev hack, is the difference between reactive debugging and a stable delivery pipeline.
Sources
- Stripe: Receive events in your webhook endpoint
- Stripe: Process undelivered webhook events
- GitHub: Best practices for using webhooks
- GitHub: Redelivering webhooks
- Webhooks
- Developer Experience
- Observability
