feat(webhook): make an outbound delivery survive a hard crash

[rmyndharis]

Webhook fan-out is fire-and-forget from the projector, so a crash between persisting a message and completing its POST lost the delivery with no record in either mode, while the documented contract promises at-least-once. With the queue off, which is the default, every in-flight and parked delivery went with it.

The in-memory inFlightDeliveries map already tracked exactly that set. This gives it a durable twin, mirroring ingress_events and its reconciler on the inbound side.

How it works

• webhook_outbox_events holds one row per (webhookId, idempotencyKey), written before the attempt. The payload is retired the moment an outcome is recorded, so only a replayable row ever carries a message body.
• The row is closed as dispatched once something durable owns the delivery: handed to BullMQ, or completed inline in direct mode. Retiring on the enqueue rather than on the POST is what stops the sweep duplicating work the queue already holds.
• A limiter rejection (dispatch capacity, shutdown drain) deliberately leaves the row pending. That is the case that used to vanish.
• WebhookReconcilerService sweeps stranded rows on an unref'd interval and replays each one through its stored idempotency key, so the retry is deduplicable at the receiver; a fresh delivery id is issued, because that names the attempt rather than the event. Overlap-guarded, batched, and bounded by a per-row budget so a stuck delivery goes terminal instead of looping.
• Settled rows are pruned daily. A pending row is never pruned on age, and the window is independent of the delivery-failure retention switch: a settled row carries no payload and no audit value, so letting an operator opt into unbounded growth buys nothing.

Five knobs, mirroring the inbound ones, documented in .env.example and forwarded through both compose files and the blank-shadow list.

What is now true, and what is not

docs/06 no longer claims a hard crash simply loses in-flight deliveries, and it names the window that is still open: the record and the message insert are not yet one transaction, so a crash between them still loses the delivery.

Two properties are recorded in the source rather than left to be rediscovered:

• The sweep takes no row claim, so two nodes on one database can both replay a delivery. The replay carries the stored idempotency key, so the cost of that race is a duplicate the contract already tells consumers to expect, rather than a lock whose holder can die mid-flight.
• A capacity-shed or drain-refused delivery keeps its failure row and is replayed, so a row in that table can belong to an event that was later delivered. It reads as a report, not a verdict.

Verification

The lint gates, all four test lanes, and an e2e that covers the crash window end to end: a delivery held open by the receiver really does leave a pending row carrying its payload, and a later instance drains it with the same idempotency key and a new delivery id.

Every assertion added here was checked by mutating the code it covers and confirming the run fails, ten mutations in total.

Writing that e2e is what caught the defect that would have shipped otherwise: the entity dated createdAt through the transformer helper, which never populated the column, so every write failed the NOT NULL constraint and was swallowed by the best-effort catch. Fourteen unit tests passed against a stubbed repository while the record silently did not exist.

Rollout

The job shape changed in the deliverOnce extraction, which shipped in 0.21.0, so this lands one release later as planned. The migration is additive and the table is new, so a rollback needs no data step.