Skip to content

fix(scheduled-executor): guarantee single execution and catch up missed runs - #663

Open
payfoxX wants to merge 1 commit into
FinChippay:mainfrom
payfoxX:issue-#632-scheduled-executor-exactly-once
Open

fix(scheduled-executor): guarantee single execution and catch up missed runs#663
payfoxX wants to merge 1 commit into
FinChippay:mainfrom
payfoxX:issue-#632-scheduled-executor-exactly-once

Conversation

@payfoxX

@payfoxX payfoxX commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

Closes #632

Description

I fixed the scheduled transaction executor so each scheduled payment is executed exactly once even when multiple workers overlap, and so missed ticks are caught up rather than silently dropped.

Problem Before

The executor polled scheduled_transactions every 60 seconds and executed any schedule whose next_run_at fell inside a ±60-second window. That left two holes:

  1. Double execution — nothing stopped two worker processes (e.g. during a deploy or a slow tick) from picking up the same due schedule and submitting it twice, a direct double-spend risk.
  2. Missed runs — a schedule whose tick was missed (executor down or a slow cycle) fell outside the ±60-second window and was never executed, so the user's payment failed silently.

Solution After

I added a claim/lease mechanism in scheduledExecutor.js so only one worker can run a given execution. The execution id is deterministic (schedule.id + next_run_at), so every worker racing on the same due run targets the same id. A worker claims it with a Redis SET NX PX lock plus a durable execution_status = 'claimed' row carrying a lease_expires_at lease; the first wins and the rest back off. The due sweep now selects everything with next_run_at <= now and skips schedules that already have a pending execution, so missed ticks are caught up on startup and on every cycle. If a worker crashes mid-execution, its lease expires and the retry sweep reclaims and completes the run. Completion flips execution_status to executed/failed and clears the lease, so a resolved run is never executed again.

Files Changed

  • backend/migrations/027_scheduled_execution_claims.js (new — adds execution_status and lease_expires_at columns)
  • backend/src/services/scheduledExecutor.js (claim/lease, Redis lock, missed-run catch-up sweep)
  • backend/tests/scheduledExecutor.test.js (regression tests + fixed pre-existing test bugs)

Tests

  • npx jest __tests__/scheduledExecutor.test.js — 23 passed
  • npx eslint (changed files) — clean
  • npx prettier --check (changed files) — clean

…ed runs

Addresses FinChippay#632. Claim each execution atomically (a Redis SET NX PX lock
plus a durable execution_status/lease row) so overlapping workers never
double-execute, reclaim expired leases so a crashed worker's run is
recovered instead of dropped, and sweep due-but-unclaimed runs
(next_run_at <= now) on startup and every tick to catch up missed runs.

🤖 Generated with Codebuff
Co-Authored-By: Codebuff <noreply@codebuff.com>
@github-actions github-actions Bot added the needs-review PR ready for Greptile AI code review label Aug 17, 2026
@github-actions

Copy link
Copy Markdown

🤖 Greptile AI Code Review

Greptile will automatically review this PR (3 file(s) changed).

Review gates:

  • ✅ CodeQL Security Scan
  • ✅ Custom rules (.greptile/config.json)
  • ✅ Architecture guidelines (.greptile/rules.md)

To manually trigger a re-review, comment @greptileai on this PR.
To skip review, add the skip-review label.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-review PR ready for Greptile AI code review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Issue #63 — Scheduled Executor Single-Execution Guarantee & Missed-Run Catch-Up

1 participant