feat(stripe): queue webhooks in dedicated FIFO worker

[charlietlamb]

Summary

• verify and enqueue Stripe webhooks before returning 200, returning 503 when the database or SQS enqueue is unavailable
• process events through a dedicated FIFO worker in per-customer order while preserving the existing handler, Stripe sync, cache refresh, and subscription lock behavior
• replace in-process early acknowledgement and Redis idempotency with FIFO deduplication and worker retry semantics

Deployment

• requires STRIPE_WEBHOOK_SQS_QUEUE_URL to point to the provisioned FIFO queue, with producer and worker IAM access

Tests

• bunx tsgo --build --noEmit
• bun test tests/unit/queue (8 passed)
• bun test tests/unit/webhooks tests/integration/scopes/scope-403.test.ts (269 passed)

---

Summary by cubic

Queues Stripe webhooks to a dedicated FIFO SQS worker that processes events per customer in order with retries and deduplication. Replaces early-ack + Redis idempotency while preserving Stripe sync and customer cache refresh.

• New Features

  • Verify and enqueue webhooks before ACK; return 503 when org/secret lookup or enqueue fails so Stripe retries.
  • FIFO worker processes by MessageGroupId (org:env:customer or account) and dedupes by event ID; sequential per group.
  • Added job StripeWebhook with longer timeouts (visibility 300s; 270s processing) and FIFO group handling.
  • Snapshots subscription locks at ingress and respects them in subscription handlers.
  • Logging, sync, and cache refresh run in worker; errors throw to trigger retries.

• Migration

  • Provision a FIFO SQS queue and set STRIPE_WEBHOOK_SQS_QUEUE_URL (must end with .fifo).
  • Grant IAM send/receive/delete to the API (producer) and worker.

^{Written for commit f13b84a. Summary will update on new commits.}

Greptile Summary

Moves verified Stripe webhooks into a dedicated FIFO SQS processing path.

• [Improvements] Enqueues verified events and processes customer groups sequentially in a dedicated worker.
• [Bug fixes] Retries webhook processing failures and preserves subscription-lock state across queue delays.
• [API changes] Returns 503 when org resolution or webhook enqueueing is unavailable and requires STRIPE_WEBHOOK_SQS_QUEUE_URL.

Confidence Score: 2/5

The PR is not safe to merge until FIFO visibility and deletion handling prevent concurrent or duplicate Stripe webhook processing.

Later messages in a customer batch can become visible while waiting behind a slow predecessor, and per-entry batch deletion failures are silently ignored, allowing the same billing events to run concurrently or more than once.

server/src/queue/processFifoMessageGroups.ts, server/src/queue/initWorkers.ts

Important Files Changed

Filename	Overview
server/src/queue/processFifoMessageGroups.ts	Adds sequential per-group FIFO processing, but queued messages can outlive their shared visibility window while waiting behind earlier handlers.
server/src/queue/initWorkers.ts	Adds the dedicated Stripe poller and FIFO dispatch, with visibility and batch-deletion behavior that permits concurrent redelivery and duplicate processing.
server/src/external/stripe/stripeWebhookQueue.ts	Verifies FIFO queue configuration and derives customer-scoped group IDs and event-scoped deduplication IDs before enqueueing.
server/src/external/stripe/processQueuedStripeWebhook.ts	Reconstructs webhook context in the worker and preserves handler, sync, cache-refresh, and logging behavior.
server/src/queue/processMessage.ts	Registers Stripe webhook dispatch and rethrows processing failures so SQS can retry them.
server/src/external/stripe/stripeWebhookRouter.ts	Replaces in-process webhook handling with verified enqueue-and-ack routing.

Sequence Diagram

sequenceDiagram
  participant Stripe
  participant API as Webhook API
  participant SQS as FIFO Queue
  participant Worker
  participant DB
  Stripe->>API: Signed webhook
  API->>API: Verify signature and resolve org
  API->>SQS: Enqueue by org/env/customer
  SQS-->>API: Accepted
  API-->>Stripe: HTTP 200
  SQS->>Worker: Receive customer message group
  loop Sequentially per group
    Worker->>DB: Load context and process event
    Worker->>DB: Refresh customer cache
  end
  Worker->>SQS: Batch-delete successful messages

Loading

Comments Outside Diff (1)

1. server/src/queue/initWorkers.ts, line 278-285 (link)

   Partial batch deletes go unnoticed

   When SQS accepts DeleteMessageBatch but returns individual entries in its Failed collection, this code discards the response and treats every deletion as successful. Those processed Stripe webhooks remain queued and are delivered again, repeating their billing mutations and downstream side effects.

   Knowledge Base Used: Queue, Cron, and Trigger.dev Background Work

   Prompt To Fix With AI

   This is a comment left during a code review.
   Path: server/src/queue/initWorkers.ts
   Line: 278-285
   
   Comment:
   **Partial batch deletes go unnoticed**
   
   When SQS accepts `DeleteMessageBatch` but returns individual entries in its `Failed` collection, this code discards the response and treats every deletion as successful. Those processed Stripe webhooks remain queued and are delivered again, repeating their billing mutations and downstream side effects.
   
   **Knowledge Base Used:** [Queue, Cron, and Trigger.dev Background Work](https://app.greptile.com/autumn-org-2/-/custom-context/knowledge-base/useautumn/autumn/-/docs/server-queue-workers.md)
   
   How can I resolve this? If you propose a fix, please make it concise.

Prompt To Fix All With AI

Fix the following 2 code review issues. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 2
server/src/queue/processFifoMessageGroups.ts:22-24
**FIFO visibility expires while waiting**

When a receive contains multiple webhooks for one customer and an earlier handler consumes a substantial part of its 270-second timeout, later messages wait in this sequential loop under the same 300-second visibility window. Their visibility can expire before processing begins, allowing another worker to process the customer group concurrently or out of order while the original worker also continues through the batch.

### Issue 2 of 2
server/src/queue/initWorkers.ts:278-285
**Partial batch deletes go unnoticed**

When SQS accepts `DeleteMessageBatch` but returns individual entries in its `Failed` collection, this code discards the response and treats every deletion as successful. Those processed Stripe webhooks remain queued and are delivered again, repeating their billing mutations and downstream side effects.

_{Reviews (1): Last reviewed commit: "feat(stripe): 🎸 queue stripe webhooks i..." | Re-trigger Greptile}

Greptile also left 1 inline comment on this PR.

Context used:

• Knowledge Base — Stripe Integration
• Knowledge Base — Queue, Cron, and Trigger.dev Background Work

[capy-ai]

Capy auto-review is paused for this organization because the usage-cycle auto-review limit has been reached. Increase the limit or turn it off in billing settings to resume automatic reviews.

[greptile-apps]

FIFO visibility expires while waiting

When a receive contains multiple webhooks for one customer and an earlier handler consumes a substantial part of its 270-second timeout, later messages wait in this sequential loop under the same 300-second visibility window. Their visibility can expire before processing begins, allowing another worker to process the customer group concurrently or out of order while the original worker also continues through the batch.

Knowledge Base Used: Queue, Cron, and Trigger.dev Background Work

Prompt To Fix With AI

This is a comment left during a code review.
Path: server/src/queue/processFifoMessageGroups.ts
Line: 22-24

Comment:
**FIFO visibility expires while waiting**

When a receive contains multiple webhooks for one customer and an earlier handler consumes a substantial part of its 270-second timeout, later messages wait in this sequential loop under the same 300-second visibility window. Their visibility can expire before processing begins, allowing another worker to process the customer group concurrently or out of order while the original worker also continues through the batch.

**Knowledge Base Used:** [Queue, Cron, and Trigger.dev Background Work](https://app.greptile.com/autumn-org-2/-/custom-context/knowledge-base/useautumn/autumn/-/docs/server-queue-workers.md)

How can I resolve this? If you propose a fix, please make it concise.