Skip to content

feat(notifications): topic-based fanout for bulk delivery#577

Open
Rocket1960 wants to merge 1 commit into
Gatheraa:mainfrom
Rocket1960:feat/notifications-fanout
Open

feat(notifications): topic-based fanout for bulk delivery#577
Rocket1960 wants to merge 1 commit into
Gatheraa:mainfrom
Rocket1960:feat/notifications-fanout

Conversation

@Rocket1960

Copy link
Copy Markdown
Contributor

Replaces the per-recipient loop in sendBulkNotifications with a single BullMQ job so the producer side goes from N DB writes to 1 queue write. The FanoutProcessor fans out at consumption time in configurable batches (default 100), using one bulk INSERT per batch to eliminate per-row lock contention.

Changes

  • notifications/queues/fanout.types.ts: job data interface, queue name, batch size constant (FANOUT_BATCH_SIZE = 100)
  • notifications/queues/fanout.processor.ts: @processor that chunks recipientIds, bulk-inserts via QueryBuilder, then delivers per-user (respects preferences, channels, WebSocket gateway, analytics)
  • notifications.service.ts: sendBulkNotifications now enqueues 1 job; return type becomes { queued, topicKey } instead of Notification[]
  • notifications.module.ts: registers BullModule + fanout queue + FanoutProcessor provider

Acceptance criteria

  • Producer: 1 BullMQ job regardless of recipient count
  • Batch INSERT: 10k recipients = 100 bulk inserts (100x fewer DB writes)
  • Scheduled fanout supported via BullMQ job delay

Closes #532

Problem: sendBulkNotifications enqueued N individual jobs and wrote N rows one-by-one — 10k attendees = 10k sequential DB inserts with row-level locks.

Fix: Producer enqueues 1 BullMQ job. FanoutProcessor fans out at consumption time in batches of 100 with a single bulk INSERT per batch → 100x fewer DB writes (exceeds ≥10x requirement).

Files changed: notifications/queues/fanout.types.ts, fanout.processor.ts, notifications.service.ts, notifications.module.ts

Replaces the per-recipient loop in sendBulkNotifications with a single
BullMQ job so the producer side goes from N DB writes to 1 queue write.
The FanoutProcessor fans out at consumption time in configurable batches
(default 100), using one bulk INSERT per batch to eliminate per-row lock
contention.

Changes
- notifications/queues/fanout.types.ts: job data interface, queue name,
  batch size constant (FANOUT_BATCH_SIZE = 100)
- notifications/queues/fanout.processor.ts: @processor that chunks
  recipientIds, bulk-inserts via QueryBuilder, then delivers per-user
  (respects preferences, channels, WebSocket gateway, analytics)
- notifications.service.ts: sendBulkNotifications now enqueues 1 job;
  return type becomes { queued, topicKey } instead of Notification[]
- notifications.module.ts: registers BullModule + fanout queue +
  FanoutProcessor provider

Acceptance criteria
- Producer: 1 BullMQ job regardless of recipient count
- Batch INSERT: 10k recipients = 100 bulk inserts (100x fewer DB writes)
- Scheduled fanout supported via BullMQ job delay
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[backend][perf] Notifications module issues fanout without batching

1 participant