|
| 1 | +--- |
| 2 | +title: Scheduler |
| 3 | +--- |
| 4 | + |
| 5 | +# Scheduler |
| 6 | + |
| 7 | +<VersionBadge version="v5.2.0" /> |
| 8 | + |
| 9 | +The scheduler is a built-in plugin that runs recurring jobs declared in a component's configuration. Harper invokes a designated export from your component on a cron or interval schedule. In a cluster, execution is leader-coordinated: under normal operation each occurrence runs once, on a single automatically elected node, rather than once per node or worker. This coordination relies on the `hdb_scheduler_state` system table replicating to every participating node (see [Cluster Behavior](#cluster-behavior)). During leadership failover, daylight-saving fall-back, or split-brain recovery an occurrence can occasionally be delivered more than once - which is why handlers must be idempotent (see below). |
| 10 | + |
| 11 | +Use it for the recurring work applications otherwise hand-roll with `setInterval`: taking daily snapshots of a dataset, pulling from an external API on a schedule, re-aggregating summary tables, generating digests or reports. |
| 12 | + |
| 13 | +For expiring old records, prefer the table-level [`expiration` and `eviction` options](../database/schema.md#table) - record cleanup is built into Harper and does not need a scheduled job. |
| 14 | + |
| 15 | +## Configuration |
| 16 | + |
| 17 | +In your component's `config.yaml`, use the `scheduler` key to declare jobs. If your application does not yet have a `config.yaml`, note that creating one **replaces** Harper's [default component configuration](./overview.md#default-configuration) rather than merging with it - keep the plugins your app relies on (`rest`, `graphqlSchema`, `jsResource`, and so on) declared alongside the `scheduler` block, or your endpoints and resources will stop loading: |
| 18 | + |
| 19 | +```yaml |
| 20 | +scheduler: |
| 21 | + jobs: |
| 22 | + - name: daily-metrics-snapshot |
| 23 | + cron: '0 2 * * *' |
| 24 | + timezone: America/New_York |
| 25 | + handler: ./jobs.js#snapshotMetrics |
| 26 | + - name: sync-exchange-rates |
| 27 | + interval: 15m |
| 28 | + handler: ./jobs.js#syncExchangeRates |
| 29 | +``` |
| 30 | +
|
| 31 | +Each job entry supports: |
| 32 | +
|
| 33 | +### `name` |
| 34 | + |
| 35 | +Type: `string` (required) |
| 36 | + |
| 37 | +A name for the job, unique within the component. Used in logs and to key the job's run state. |
| 38 | + |
| 39 | +### `cron` |
| 40 | + |
| 41 | +Type: `string` |
| 42 | + |
| 43 | +A five-field cron expression: minute, hour, day-of-month, month, day-of-week. Supports `*`, lists (`1,15`), ranges (`9-17`), steps (`*/15`, `0-59/5`), month and day names (`JAN`, `MON-FRI`), and the common macros (`@hourly`, `@daily`, `@midnight`, `@weekly`, `@monthly`, `@yearly`). Day-of-week `0` and `7` both mean Sunday. Following the POSIX cron rule, when both day-of-month and day-of-week are restricted, the job runs when either matches. |
| 44 | + |
| 45 | +There is no seconds field. Expressions that can never fire (such as `0 0 30 2 *` - February 30th) are rejected when the component loads. |
| 46 | + |
| 47 | +Always quote the expression in YAML: a leading `*` (as in `*/5 * * * *`) is YAML alias syntax and `@daily` starts with a reserved character - unquoted, both fail with confusing YAML parse errors rather than anything about cron. |
| 48 | + |
| 49 | +Exactly one of `cron` or `interval` is required. |
| 50 | + |
| 51 | +### `interval` |
| 52 | + |
| 53 | +Type: `string` (duration) or `number` (seconds) |
| 54 | + |
| 55 | +A simple cadence instead of a cron expression: a number of seconds, or a duration string like `90s`, `5m`, `1h`, `1d`. Must be between one second and 365 days. Use this for "every N" maintenance passes that do not align to wall-clock times. |
| 56 | + |
| 57 | +### `timezone` |
| 58 | + |
| 59 | +Type: `string` |
| 60 | + |
| 61 | +An IANA timezone (for example `America/Chicago`) the cron expression is evaluated in. Defaults to the server's timezone - specifically, the local timezone of whichever node is currently the scheduler leader. In a cluster whose nodes may not share a host timezone, set an explicit `timezone`: otherwise a failover between differently-configured nodes can shift a job like `0 2 * * *` by hours, producing an extra or missing wall-clock run. Only valid with `cron`. |
| 62 | + |
| 63 | +During daylight-saving transitions: a job scheduled inside the spring-forward gap (a wall-clock time that does not exist that day) runs at the shifted instant rather than being skipped, and a job scheduled inside the fall-back overlap (a wall-clock time that occurs twice) runs once, at the first occurrence. |
| 64 | + |
| 65 | +### `handler` |
| 66 | + |
| 67 | +Type: `string` (required) |
| 68 | + |
| 69 | +The module and export to invoke, as `<module path>#<named export>`, relative to the component directory - for example `./jobs.js#snapshotMetrics`. Omit the `#...` suffix to use the module's default export. Do not put a space before the `#` (YAML would treat the rest of the line as a comment and silently drop the export name). The module is loaded in your component's environment, so it has access to the same globals (`tables`, `databases`, `server`, and so on) as the rest of your component code. |
| 70 | + |
| 71 | +A bad handler reference (missing module, missing export, non-function export) fails the component load at deploy time rather than failing silently at the first scheduled fire. |
| 72 | + |
| 73 | +## Writing a Handler |
| 74 | + |
| 75 | +The handler is called with a single context argument and may return a promise: |
| 76 | + |
| 77 | +```javascript |
| 78 | +// Nightly snapshot: roll the current state of a table into a dated snapshot record |
| 79 | +export async function snapshotMetrics(context) { |
| 80 | + // context.jobName - the configured job name |
| 81 | + // context.scheduledAt - the occurrence this run is for (Date) |
| 82 | + // context.catchUp - true when this run is making up a missed occurrence |
| 83 | + const day = context.scheduledAt.toISOString().slice(0, 10); |
| 84 | + let total = 0; |
| 85 | + let active = 0; |
| 86 | + for await (const device of tables.Devices.search({})) { |
| 87 | + total++; |
| 88 | + if (device.status === 'active') active++; |
| 89 | + } |
| 90 | + // Keyed by day, so a duplicate delivery of the same occurrence just rewrites |
| 91 | + // the same record - this is what makes the handler idempotent |
| 92 | + await tables.DailyDeviceSnapshot.put({ id: day, total, active }); |
| 93 | +} |
| 94 | +
|
| 95 | +// Scheduled pull from an external API into a Harper table |
| 96 | +export async function syncExchangeRates(context) { |
| 97 | + // Always bound external calls: a request that never settles would leave this |
| 98 | + // single-flight job stuck (runs never overlap, so nothing reschedules while |
| 99 | + // a run is in flight) |
| 100 | + const response = await fetch('https://api.example.com/rates', { signal: AbortSignal.timeout(10_000) }); |
| 101 | + if (!response.ok) throw new Error(`rates API responded ${response.status}`); |
| 102 | + const payload = await response.json(); |
| 103 | + // Persist only the fields you expect - Harper schemas are flexible by |
| 104 | + // default, so an unvalidated spread would store whatever shape the |
| 105 | + // upstream happened to send |
| 106 | + const { usd, eur, gbp } = payload; |
| 107 | + if (typeof usd !== 'number' || typeof eur !== 'number' || typeof gbp !== 'number') { |
| 108 | + throw new Error('rates API returned an unexpected shape'); |
| 109 | + } |
| 110 | + await tables.ExchangeRates.put({ usd, eur, gbp, id: 'latest', fetchedAt: context.scheduledAt }); |
| 111 | +} |
| 112 | +``` |
| 113 | + |
| 114 | +**Handlers should be idempotent.** Harper's clustering has no distributed lock, so leadership failover and daylight-saving fall-back can occasionally deliver the same logical occurrence twice. Design handlers so that running twice for one occurrence is harmless. |
| 115 | + |
| 116 | +**A failed run is not retried.** If a handler throws (or its outbound request times out), the run is recorded with `lastStatus: error` and the occurrence is not made up - the job simply fires again at its next scheduled time. Handlers that need delivery guarantees should implement their own bounded retry/backoff inside the handler. |
| 117 | + |
| 118 | +For a single, unchanged job registration on one scheduler leader, runs never overlap. Two boundaries pierce that guard, and both mean idempotency must cover concurrent execution, not just repeated execution: across nodes, the rare duplicate-delivery cases above (a partitioned leader with a slow handler still in flight while its successor promotes) run one occurrence on two nodes at once; and across reloads, a redeploy cannot cancel a handler that is already executing, so the redeployed job can fire while the previous registration's run is still in flight on the same node. Missed time is not queued up, but it is also not always skipped outright: when a run outlasts its cadence, an interval job's next run starts promptly after the previous one finishes (a slow handler can therefore run back-to-back), and a cron job makes up at most its single most recent missed occurrence (delivered with `context.catchUp` set to `true`). |
| 119 | + |
| 120 | +## Cluster Behavior |
| 121 | + |
| 122 | +One node in the cluster - the scheduler leader - runs all scheduled jobs; the others watch. Leader election is automatic and requires no configuration: |
| 123 | + |
| 124 | +- The leader maintains a heartbeat in the replicated `system` database. If it stops heartbeating (crash, shutdown, partition) for more than five minutes, the next node in line promotes itself; with default timings, failover completes within about six and a half minutes when the first eligible successor is up. Each earlier successor that is also down adds a further escalation delay (two and a half minutes per node) before the next one in line claims leadership. |
| 125 | +- A restarting node defers to an actively heartbeating leader, so leadership is stable across deploys and restarts. |
| 126 | +- On a single-node instance, that node simply runs the jobs. |
| 127 | + |
| 128 | +### Missed Occurrences |
| 129 | + |
| 130 | +If the most recent occurrence of a cron job was missed - the leader was down, a failover was in progress, or daylight-saving time skipped the slot - the scheduler fires one catch-up run for it (with `context.catchUp` set to `true`). Only the single most recent missed occurrence is made up, not every occurrence missed during an outage. Interval jobs resume their cadence from their last recorded run. |
| 131 | + |
| 132 | +A newly deployed job waits for its first scheduled time; it does not fire immediately on deploy. |
| 133 | + |
| 134 | +### Replication Topology |
| 135 | + |
| 136 | +The cluster-once guarantee holds only where the `hdb_scheduler_state` system table actually replicates: leader election and run-state coordination happen through its rows. On constrained or directional replication topologies where the `system` database does not reach every node, nodes that cannot see the leader's heartbeat will each elect themselves and run every job continuously - not just the occasional failover duplicate that idempotency covers. If you scope replication (for example with `replication.databases`), ensure the `system` database replicates among all scheduler-participating nodes. |
| 137 | + |
| 138 | +### Run State |
| 139 | + |
| 140 | +Each job's last run time, status, duration, and last error (if any) are recorded in the `hdb_scheduler_state` system table, alongside the leader lease. This state replicates across the cluster so a newly promoted leader knows what has already run. |
0 commit comments