fix(messagequeue): run queue garbage collection on busy partitions - #622
fix(messagequeue): run queue garbage collection on busy partitions#622Jal-Bafana wants to merge 2 commits into
Conversation
|
|
There was a problem hiding this comment.
Pull request overview
This PR fixes MySQL-backed message queue garbage collection (GC) starvation on continuously busy partitions by running the existing GC cadence on every poll tick (still throttled), and adds regression tests to prevent reintroduction.
Changes:
- Run per-partition GC every N poll ticks regardless of whether messages were delivered.
- Add a unit test ensuring GC triggers even when every tick delivers a message.
- Add a MySQL integration test verifying acked rows are reclaimed under sustained traffic.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| test/integration/extension/messagequeue/mysql/queue_test.go | Adds an integration regression test validating GC reclaims acked rows during continuous traffic. |
| platform/extension/messagequeue/mysql/subscriber.go | Changes GC scheduling from “idle ticks only” to “every N poll ticks” while preserving throttling. |
| platform/extension/messagequeue/mysql/subscriber_test.go | Adds a unit test proving GC runs on busy ticks (no idle periods). |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
There was a problem hiding this comment.
Two notes on the diff — the core change looks right to me. Removing the messageCount == 0 gate genuinely fixes the starvation (the old else branch reset the counter to 0 forever on a continuously busy partition), GC load stays bounded the same way it was, and run() only logs pollAndDeliver errors so a GC failure can't kill a worker or cause re-delivery.
- Record messages_delivered before the GC block so a GC failure can no longer drop the counter for already-delivered messages on busy ticks. - Seed the integration backlog before subscribing so the GC counter starts at zero when polling begins and cannot fire mid-drain. - Drain hook signals inside the traffic loop so the worker's blocking signal send cannot fill the buffer and stall the partition.
Why?
Garbage collection of acknowledged
queue_messagesrows currently runs only on idle poll ticks.A continuously busy partition resets the GC counter whenever it delivers a message, so it can never reach the GC threshold. Under sustained traffic, acknowledged message rows can therefore accumulate indefinitely.
This also means the message deduplication horizon can remain unnecessarily large on busy partitions.
What?
Run the existing garbage collection cadence on every poll tick rather than only idle ticks.
The existing 100-tick throttle is preserved, so the per-partition GC frequency is not increased. The GC safety mechanism is also unchanged: garbage collection still uses the minimum acknowledged offset across consumer groups.
Added:
Testing
go test ./platform/extension/messagequeue/...— PASSgo test ./platform/base/...— PASSgo veton the messagequeue and integration packages — PASSgofmtandgit diff --check— cleanNot run locally due to environment limitations:
TestGCReclaimsAckedRowsUnderContinuousTraffic— Docker daemon unavailablemake gazelle,make fmt,make lint,make check-tidy,make check-gazelleand Bazel tests —makeunavailable and Bazel's Windows C++ toolchain is not configured