feat(notifications): topic-based fanout for bulk delivery#577
Open
Rocket1960 wants to merge 1 commit into
Open
feat(notifications): topic-based fanout for bulk delivery#577Rocket1960 wants to merge 1 commit into
Rocket1960 wants to merge 1 commit into
Conversation
Replaces the per-recipient loop in sendBulkNotifications with a single BullMQ job so the producer side goes from N DB writes to 1 queue write. The FanoutProcessor fans out at consumption time in configurable batches (default 100), using one bulk INSERT per batch to eliminate per-row lock contention. Changes - notifications/queues/fanout.types.ts: job data interface, queue name, batch size constant (FANOUT_BATCH_SIZE = 100) - notifications/queues/fanout.processor.ts: @processor that chunks recipientIds, bulk-inserts via QueryBuilder, then delivers per-user (respects preferences, channels, WebSocket gateway, analytics) - notifications.service.ts: sendBulkNotifications now enqueues 1 job; return type becomes { queued, topicKey } instead of Notification[] - notifications.module.ts: registers BullModule + fanout queue + FanoutProcessor provider Acceptance criteria - Producer: 1 BullMQ job regardless of recipient count - Batch INSERT: 10k recipients = 100 bulk inserts (100x fewer DB writes) - Scheduled fanout supported via BullMQ job delay
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces the per-recipient loop in sendBulkNotifications with a single BullMQ job so the producer side goes from N DB writes to 1 queue write. The FanoutProcessor fans out at consumption time in configurable batches (default 100), using one bulk INSERT per batch to eliminate per-row lock contention.
Changes
Acceptance criteria
Closes #532
Problem: sendBulkNotifications enqueued N individual jobs and wrote N rows one-by-one — 10k attendees = 10k sequential DB inserts with row-level locks.
Fix: Producer enqueues 1 BullMQ job. FanoutProcessor fans out at consumption time in batches of 100 with a single bulk INSERT per batch → 100x fewer DB writes (exceeds ≥10x requirement).
Files changed: notifications/queues/fanout.types.ts, fanout.processor.ts, notifications.service.ts, notifications.module.ts