fix(scheduled-executor): guarantee single execution and catch up missed runs - #663
Open
payfoxX wants to merge 1 commit into
Open
fix(scheduled-executor): guarantee single execution and catch up missed runs#663payfoxX wants to merge 1 commit into
payfoxX wants to merge 1 commit into
Conversation
…ed runs Addresses FinChippay#632. Claim each execution atomically (a Redis SET NX PX lock plus a durable execution_status/lease row) so overlapping workers never double-execute, reclaim expired leases so a crashed worker's run is recovered instead of dropped, and sweep due-but-unclaimed runs (next_run_at <= now) on startup and every tick to catch up missed runs. 🤖 Generated with Codebuff Co-Authored-By: Codebuff <noreply@codebuff.com>
🤖 Greptile AI Code ReviewGreptile will automatically review this PR (3 file(s) changed). Review gates:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #632
Description
I fixed the scheduled transaction executor so each scheduled payment is executed exactly once even when multiple workers overlap, and so missed ticks are caught up rather than silently dropped.
Problem Before
The executor polled
scheduled_transactionsevery 60 seconds and executed any schedule whosenext_run_atfell inside a ±60-second window. That left two holes:Solution After
I added a claim/lease mechanism in
scheduledExecutor.jsso only one worker can run a given execution. The execution id is deterministic (schedule.id + next_run_at), so every worker racing on the same due run targets the same id. A worker claims it with a RedisSET NX PXlock plus a durableexecution_status = 'claimed'row carrying alease_expires_atlease; the first wins and the rest back off. The due sweep now selects everything withnext_run_at <= nowand skips schedules that already have a pending execution, so missed ticks are caught up on startup and on every cycle. If a worker crashes mid-execution, its lease expires and the retry sweep reclaims and completes the run. Completion flipsexecution_statustoexecuted/failedand clears the lease, so a resolved run is never executed again.Files Changed
Tests
npx jest __tests__/scheduledExecutor.test.js— 23 passednpx eslint(changed files) — cleannpx prettier --check(changed files) — clean