Skip to content

Keep running jobs alive across dispatcher and task-pod restarts (job monitor lease) - #1198

Merged
cigamit merged 30 commits into
mainfrom
job-monitor-lease
Nov 11, 2026
Merged

cigamit merged 30 commits into
mainfrom
job-monitor-lease

Conversation

@cigamit

@cigamit cigamit commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Running jobs now survive a restart or crash of the dispatcher that monitors them, including the replacement of a task pod on Kubernetes.

Today a graceful stop cancels the job's receptor work unit and fails the job with "Task was canceled due to receiving a shutdown signal." For a container group, cancelling the unit also deletes the job's pod. A crash ends the same way: the restarted dispatcher, or a peer's heartbeat, reaps the job as failed, and the pod reaper then deletes its pod.

With this change the automation keeps running. Another control node, or the same one once it is back, picks up reading the job's output from where it survived. The job then finishes normally, with its complete output.

This is an alternative to the cross-controller job adoption that upstream AWX is building in ansible/awx#16669 and the stacked drafts #16728 to #16734. It reuses several of their pieces, but it decides who reads a job with a lease on the job rather than from the health of its controller. The comparison is under "How this differs from upstream" below.

Design notes for reviewers and operators are in docs/job_monitor_lease.md.

How it works

flowchart TD
    A["Job handed to receptor"] --> B["Launching task reads the output and renews the lease every 10 s"]
    B -->|"output read"| C["Lease removed, then post-run hooks and final status"]
    B -->|"shutdown, worker started"| D["Detach: the work keeps running and the lease is expired at once"]
    B -->|"shutdown, worker had no payload yet"| R["Cancel the unit and re-queue the job"]
    B -->|"dispatcher killed"| E["Lease lapses after 60 s"]
    D --> S["Sweep on any ready control node, every 30 s: claims the job by compare-and-set on celery_task_id"]
    E --> S
    S --> P{"Where did the output survive?"}
    P -->|"pod log or receptor unit"| Q["Read it from the start, skipping events already saved"]
    Q --> C
    P -->|"not reachable yet"| W["Put off 30 s, at most 120 times"]
    W --> S
    P -->|"gone, and nothing ran or the job repeats safely"| R
    P -->|"gone"| F["Fail the job with the reason"]
Loading

The lease. JobMonitorLease is a new table with one row per job whose output is being read. The task that launched the job creates the row once receptor has the whole payload. It renews the row every JOB_MONITOR_RENEW_SECONDS while it reads, and removes it as soon as the output has been read. That happens before the post-run hooks, which can run longer than a lease lasts. The row also records two facts a later reader needs:

  • which pod a container group job runs in;
  • whether ansible-runner reported starting (worker_started), meaning the execution environment received its payload.

Shutdown or cancel. UnifiedJob.cancel() commits the job's cancel_flag before it signals the dispatcher, so the flag tells a user's cancel from a shutdown. On a shutdown, the task reading a job:

  • detaches if the worker has its payload: it stops reading, leaves the unit and pod running, and expires its own lease;
  • otherwise cancels the unit and re-queues the job, since nothing has run yet (a unit that cannot be cancelled is left to the next monitor instead);
  • always cancels and re-queues a project update, which runs under an exclusive project lock held by its task. An inline project sync is only cancelled, because the job that started it is stopping too.

The sweep. sweep_job_monitors runs every 30 seconds on every ready control node. It claims each expired lease of a running job with one compare-and-set on the job's celery_task_id, so two nodes cannot both take a job. The winner sets itself as controller_node and dispatches resume_job under the new task id. A node may claim:

  • its own jobs, always, since they already count against its capacity;
  • another node's jobs only when it is enabled and has control capacity to spare.

Work that ran in a control or hybrid node's own receptor can be read only through that node. While that node is ready, or restarting its dispatcher, nobody else claims the job. The sweep also deletes expired leases of jobs that ended while nobody was reading them.

Resuming. BaseTask.resume() asks plan_resume() where the output survived:

The job ran Its output is read from
in a container group pod the Kubernetes pod log API (PodLogStream). A running pod is followed, reconnecting from the last timestamp; a finished pod is read once.
through a unit this node's receptor still has (only the dispatcher restarted) work results (ReceptorUnitStream)
on an execution node, or in a container group behind a mesh node work adopt, then work results. This needs a receptor newer than 1.6.9; see Limits.
in a unit on a control node that is gone nowhere. The job is re-queued if nothing ran or it repeats safely (project and inventory updates), and failed otherwise.

Some cases need waiting or cleanup:

  • A pending pod is waited for, up to ASCENDER_CONTAINER_GROUP_POD_PENDING_TIMEOUT.
  • A pod whose launch died before the worker started, and which has written nothing for JOB_MONITOR_WEDGE_SECONDS, never got its payload. It is deleted and the job is re-queued.
  • Output that is not reachable yet, or a transient Kubernetes or receptor error, puts the job off by JOB_MONITOR_RETRY_SECONDS, at most JOB_MONITOR_MAX_DEFERRALS times. Waiting for a pending pod does not count toward that limit; the pending timeout bounds it.
  • An execution or hop node that is unavailable, but not yet declared lost, may still be running the unit: the job waits for it, counted against the same limit.

Replaying without duplicates. The stream goes to ansible-runner's process streamer unchanged, from the beginning, so events are parsed, saved and broadcast exactly as during a launch. Replaying from the start also recovers events that were still queued in a lost control pod's valkey.

  • RunnerCallback skips counters that are already saved, read as ranges together with their count in one query.
  • A new unique constraint on (job, job_created, counter) on all five event tables is the guarantee. It replaces the plain index on the same columns, so inserts maintain no more indexes than before.
  • The callback receiver drops a duplicate instead of retrying it.
  • Host summaries, host metrics and the stats notification are written once.
  • finished is taken from the wrapup event, so a job that ended while nobody was watching is not stamped late.

Fencing. The final status write uses update_model(owner_task_id=...). It locks the row and raises NotOwner if celery_task_id has moved on. A reader whose job was claimed by another node finds out at its next renewal (LeaseLost). It then stops without finalizing or releasing anything. A project update also stops its work, since it never runs without its task's lock on the project, and a job whose inline sync this was fails with it. Claims and retirements serialize on the lease row: a claim re-checks under the row's lock that the lease is still expired, and a reader retires its lease under the same lock before it releases the unit or pod, so neither can act on a job the other has just taken or finished. Expiring and putting off a lease are fenced by the owner's task id as well.

Re-queued jobs keep their id and start over from pending. The abandoned attempt's events are deleted when the job is re-queued, and again just before the new attempt is submitted. A job launched with passwords prompted at launch is failed instead, because those passwords are cleared from the job when it starts.

Credentials. A resume never overwrites the job_env the launch stored, which has the injected credential values masked. A resume cannot rebuild that masking. If the launch stored no environment, the resume stores one with every value a credential could have set hidden.

Reapers. startup_reaping, the heartbeat's reap(), the pool's dead-worker cleanup and the task manager's orphan reaper leave a leased job to the sweep when the node in question is the job's controller; the heartbeat's reap() of a control or hybrid node, which executes only work it controls, leaves all its leased jobs to the sweep. A leased job whose execution or hop node is lost is reaped as before: its controller is still reading it, and receptor would wait for that node forever.

How this differs from upstream

Upstream infers that a job has lost its reader from the health of the job's controller, and then decides who adopts it. Here the reader states it directly by renewing a lease on the job. An expired lease is the only trigger, and one compare-and-set on the job row is the only arbiter.

Upstream work as of 2026-11-10 (none of it merged):

  • AAP-89602: implement cross-controller job adoption ansible/awx#16669 (open; +8,150/−292 in 28 files, about 2,200 lines of it product code). A surviving controller adopts the running jobs of a controller it judges lost from heartbeats. The PR covers:

    • adoption at startup and from the heartbeat, plus an orphan sweep;
    • a shutdown announcement broadcast to peers and a startup probe of peer pods;
    • stall budgets kept in the Django cache, wedged-pod detection and container-group pod triage;
    • work adopt for mesh units and a status-based fence on the final write.

    It needs receptor #1564, #1605, #1606 and #1607. These are merged to receptor devel but are in no release; 1.6.9 has none of them. It also relies on dispatcherd behaviour such as on_duplicate='discard', which Ascender's dispatcher does not have.

  • #16728 to #16734 (drafts, stacked on devel rather than on #16669). They make the heartbeat's adoption decisions testable: a pure decision module, a fake receptor and an injectable clock. #16731 adds a second fence, claim-then-dispatch with owner_task_id/NotOwner. #16733 and #16734 are tests only.

    • Upstream's failure testing on a two-controller cluster found 29 defects in devel's existing adoption, and #16733 reproduces 22 of them.
    • #16734 reports that adding an orphan sweep like #16669's, which adopts any job whose controller is not ready, fails four of its five live-controller guard tests.
Upstream This PR
What makes a job need a new reader Its controller looks lost, judged from heartbeats, lost-instance handling, startup checks, an orphan sweep and a shutdown broadcast Its lease expired. No heartbeat, clock or mesh state is consulted.
Ways a job can change hands Several: startup, heartbeat, orphan sweep, shutdown One sweep and one resume task
Claim and fence Two designs in flight: status-based in #16669, owner_task_id/NotOwner in #16731 Compare-and-set claim and NotOwner, as in #16731, plus LeaseLost so that a stale reader stops at its next renewal; claims and retirements serialize on the lease row
Running container-group pods A new receptor unit attached to the existing pod (pod_name, receptor devel only) The Kubernetes pod log API, for running and finished pods alike, with no receptor change
Execution and hop nodes work adopt (receptor devel) Same
"The payload never arrived" Inferred from silence Recorded on the lease (worker_started). Silence is used only for a pod whose launch died before starting.
Retry budget Django cache deferrals column on the lease row
Stored environment of a resumed job Rebuilt with build_safe_env, which masks by variable name only, so a custom credential value under an ordinary name is stored in clear text The launch's masked job_env is kept. Without one, every value a credential could have set is hidden.
Dispatcher Relies on dispatcherd Ascender's dispatcher, changed only in the reaper hooks
Duplicate events Unique (job, counter, job_created) constraint, receiver dedup and a skip in the callback Same approach, ported. The constraint replaces the old plain index instead of adding a second one.
Size #16669: ~2,200 product, ~5,500 test and ~480 doc lines. Drafts: ~550 product and ~2,300 test lines. ~2,450 product, ~130 migration, ~2,300 test and ~200 doc lines

The following are ported from upstream: replay from the start with counter skipping, the unique constraint, receiver and host-summary dedup, finished from the wrapup event, the host map rebuilt from inventory data (#16669), and the compare-and-set claim with the NotOwner fence (#16731). The scenario lists in #16733 and #16734 served as a review checklist.

What changes for users and operators

  • Stopping or restarting a dispatcher or task pod no longer fails running jobs.

    • After a graceful stop, another ready control node picks them up at its next sweep, within 30 seconds.
    • A node on its own picks its jobs back up once its dispatcher is running again: about 50 seconds in the runs below.
    • After a crash, the lease must lapse first, so it takes about 60 to 90 seconds.

    The automation runs throughout; only the saving of its events waits.

  • A resumed job says so in job_explanation: "Output monitoring was resumed on <node> after the node reading it stopped."

  • A job that had not started running, and any project update, is re-queued instead of failed: "The job was re-queued because …". The existing pre_start clears that explanation when the job starts again.

  • There is a new periodic task job_monitor_sweep (30 s), and there are new lifecycle events: monitor_leased, pod_recorded, monitor_resumed, monitor_detached, monitor_requeued, monitor_resume_failed and monitor_resume_canceled.

  • The API and UI are unchanged. The lease table is internal.

  • Kubernetes: reading a pod's log needs get on pods/log. receptor already needs that permission for container groups, so no operator or RBAC change is needed. As a follow-up, the operator's long termination grace period can be shortened once this ships.

New settings (all optional)
Setting Default Meaning
JOB_MONITOR_LEASE_SECONDS 60 How long a lease holds without a renewal. A crash is noticed within this.
JOB_MONITOR_RENEW_SECONDS 10 How often a reader renews its lease and checks for signals
JOB_MONITOR_SWEEP_BATCH 50 Most expired leases one sweep looks at
JOB_MONITOR_RETRY_SECONDS 30 How long a resume puts a job off when its output is not reachable yet or could not be read
JOB_MONITOR_MAX_DEFERRALS 120 How many such waits before the job is failed
JOB_MONITOR_WEDGE_SECONDS 60 Silence after which a pod that never got its payload is deleted and its job re-queued
JOB_MONITOR_STREAM_IDLE_SECONDS 120 Silence after which a pod log connection is reopened, or a finished unit's short stream is abandoned

Upgrade notes

  • 0232_event_counter_unique checks each event table for duplicate (job, job_created, counter) rows and deletes the later copies; a healthy database has none. It then adds the unique constraints and drops the plain indexes they replace. On large event tables this costs about as much as reindexing each partition, under a lock on each table.
  • 0233_job_monitor_lease creates the new table.
  • Jobs already running during the upgrade have no lease and are handled exactly as today.

Limits

  • The kubelet serves only a container's current log file. A job that writes more than containerLogMaxSize (10 MiB by default) while unmonitored loses the unsaved events in between. Its final artifacts are always in the current file.
  • Work in a control node's own receptor, meaning project updates and jobs on hybrid nodes, cannot be read once that node is gone. Such a job is re-queued when that is safe and failed otherwise.
  • The post-run hooks are not resumed. If the task running them dies, the job fails as it does today.
  • System jobs, and jobs not yet handed to receptor, end on a shutdown as they do today.
  • Execution and hop node adoption needs work adopt (Add work adopt command to track a remote work unit ansible/receptor#1564, not in 1.6.9). It is covered only by tests with a fake receptor. With a receptor without it, a job whose controller is lost while the job runs on an execution node is retried until JOB_MONITOR_MAX_DEFERRALS and then failed, about an hour at the defaults. Today such a job fails within minutes.
  • Each lease renewal closes its database connection, so a long job does not hold an idle connection open. The cost is one reconnect per running job every 10 seconds.

Testing

  • Full suite: 5,422 passed, 7 skipped. The one warning is an existing pagination warning in the workflow tests. check_migrations and check_settings are clean, and so are ruff check and ruff format --check.
  • New tests:
    • tests/functional/test_job_monitor.py (109 tests): lease fencing, the sweep's claim rules, every plan_resume outcome, resume, shutdown handling per job type, the reapers and re-queueing.
    • tests/unit/tasks/test_streams.py (35 tests): pod log reconnects and timeouts, connection close order, receptor stream stall detection.
  • The stdout API tests now give their events distinct counters, as the callback receiver always has.
  • Live runs used the docker-compose dev environment, with a kind cluster (Kubernetes 1.37) as the container group. The playbook ran 24 steps and produced 36 events.
Scenario Result
Graceful dispatcher restart, job on the local node Resumed about 50 s later, successful, 36/36 events in order
Graceful dispatcher restart, container group pod Resumed from the pod log about 50 s later, successful, 36/36, pod removed at the end
SIGKILL to the dispatcher process group Lease lapsed, resumed about 80 s later, successful, 36/36
Dispatcher stopped for 150 s while the pod finished Result read from the completed pod, finished back-dated to the pod's end, 36/36, pod removed
Graceful dispatcher restart, container group pod, with a 3 s stream idle timeout so the resumed pod log reconnects at every 5 s pause Resumed once, no deferrals, successful, 36/36 events in order
Restart during a project update from GitHub Cancelled and re-queued; the re-run was successful with 43 events in order
Dispatcher killed while the pod was unschedulable (pending), then the job cancelled The resume waited on the pending pod. The cancel ended the job canceled at once, and the lease and pod were gone within a minute.
Where to start reviewing
  • ascender/main/models/monitor.py: the lease model and its manager.
  • ascender/main/tasks/monitor.py: the sweep, the claim, plan_resume and re-queueing.
  • ascender/main/tasks/receptor.py: lease start and renewal in AWXReceptorJob, shutdown handling (_handle_signal), and ReceptorUnitStream, including work adopt.
  • ascender/main/tasks/streams.py: PodLogStream.
  • ascender/main/tasks/jobs.py: BaseTask.resume(), the fenced finish_run, and clearing an abandoned attempt's events before submitting.
  • ascender/main/tasks/callback.py: the replay skip, the host map on resume, and job_env on resume.
  • ascender/main/dispatch/worker/callback.py, ascender/main/models/events.py and migration 0232: duplicate handling.
  • ascender/main/utils/update_model.py: NotOwner. Retries now keep select_for_update, which they used to drop.
  • ascender/main/dispatch/reaper.py, ascender/main/dispatch/pool.py and ascender/main/scheduler/task_manager.py: the reapers.
Trying it on the dev environment
  1. Launch a job template whose playbook runs for a few minutes, for example a loop of pause: seconds=5.
  2. While it runs, run docker exec tools_ascender_1 supervisorctl restart ascender-processes:dispatcher, or send SIGKILL to the dispatcher's process group.
  3. The job stays running. Its output continues in the UI about a minute after a restart, or a minute and a half after a kill. It ends successful, with events numbered from 1 without gaps and the "Output monitoring was resumed" explanation.
  4. For a container group, watch kubectl get pods as well. The restart does not touch the automation pod, and the pod is removed when the job finishes.

@cigamit
cigamit requested a max review from Copilot November 10, 2026 20:34
@cigamit cigamit self-assigned this Nov 10, 2026
@cigamit cigamit added the enhancement New feature or request label Nov 10, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved critical lease-ownership and recovery flaws can rerun partially executed jobs, lose output, and leave previous executions running after finalization.

22 open issues

And 2 more that still need to be addressed.

What changed in this PR

Introduces lease-based recovery in Ascender’s task engine so running automation can continue while another dispatcher resumes monitoring after a restart.

Changes:

  • Adds monitor leases, recovery planning, shutdown detachment, and ownership-fenced completion.
  • Resumes output from Kubernetes pods or receptor units, with replay deduplication and updated reaper behavior.
  • Adds migrations, recovery tests, configurable settings, and operator documentation.
File Description
docs/​job_monitor_lease.md Documents recovery behavior, limitations, and upgrades.
ascender/​settings/​typed.py Adds types for monitor settings.
ascender/​settings/​defaults.py Defines defaults and schedules the monitor sweep.
ascender/​main/​utils/​update_model.py Adds ownership fencing and preserves locking during retries.
ascender/​main/​tests/​unit/​tasks/​test_streams.py Tests pod and receptor stream behavior.
ascender/​main/​tests/​functional/​test_job_monitor.py Tests leases, claims, recovery, and reapers.
ascender/​main/​tests/​functional/​api/​test_unified_jobs_stdout.py Gives stdout fixtures distinct event counters.
ascender/​main/​tasks/​streams.py Adds reconnectable Kubernetes pod-log streaming.
ascender/​main/​tasks/​receptor.py Adds lease renewal, unit adoption, and shutdown handling.
ascender/​main/​tasks/​monitor.py Implements sweeping, claiming, recovery planning, and re-queueing.
ascender/​main/​tasks/​jobs.py Adds resume handling and shared, fenced finalization.
ascender/​main/​tasks/​callback.py Adds replay filtering and preserves masked environments.
ascender/​main/​tasks/​__init__.py Registers monitor tasks.
ascender/​main/​scheduler/​task_manager.py Adjusts orphan reaping for leased jobs.
ascender/​main/​models/​monitor.py Defines the lease model and manager.
ascender/​main/​models/​events.py Adds event uniqueness and replay-aware host summaries.
ascender/​main/​models/​__init__.py Exports lease-related classes.
ascender/​main/​migrations/​0233_job_monitor_lease.py Creates the monitor lease table.
ascender/​main/​migrations/​0232_event_counter_unique.py Deduplicates events and replaces indexes with unique constraints.
ascender/​main/​dispatch/​worker/​callback.py Drops duplicate events and guards stats notifications.
ascender/​main/​dispatch/​reaper.py Leaves leased controller jobs to monitor recovery.
ascender/​main/​dispatch/​pool.py Releases dead workers’ leases for recovery.

Review effort: Max

Comment thread ascender/main/models/events.py Resolved
Comment thread ascender/main/models/monitor.py Resolved
Comment thread ascender/main/scheduler/task_manager.py Resolved
Comment thread ascender/main/tasks/callback.py Resolved
Comment thread ascender/main/tasks/callback.py Resolved
Comment thread ascender/main/tasks/streams.py Resolved
Comment thread ascender/main/tasks/streams.py Resolved
Comment thread ascender/main/tasks/streams.py Resolved
Comment thread ascender/main/tasks/monitor.py Resolved
Comment thread ascender/main/tasks/monitor.py Resolved
Copilot AI max review requested due to automatic review settings November 10, 2026 20:59

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved critical lease-ownership races and replay-handling defects can strand jobs, duplicate execution, and lose events or completion notifications.

32 open issues

And 12 more that still need to be addressed.

Review effort: Max

Comment thread ascender/main/models/monitor.py Resolved
Comment thread ascender/main/tasks/callback.py Resolved
Comment thread ascender/main/tasks/callback.py Resolved
Comment thread ascender/main/tasks/monitor.py Resolved
Comment thread ascender/main/tasks/monitor.py Resolved
Comment thread ascender/main/tasks/receptor.py Resolved
Comment thread ascender/main/tasks/streams.py Resolved
Comment thread ascender/main/tasks/streams.py Resolved
Comment thread ascender/main/tasks/receptor.py Resolved
@2and3makes23

Copy link
Copy Markdown
Contributor

This addresses imho one of the biggest weaknesses of AWX - very much looking forward to this! 😍

cigamit and others added 18 commits November 10, 2026 16:46
The generated kubernetes client (36.0.3) leaves sinceTime out of
read_namespaced_pod_log and rejects since_time as an unexpected keyword
argument, so every reconnect of a pod log after the first line raised
ApiTypeError: a followed log that stayed silent for
JOB_MONITOR_STREAM_IDLE_SECONDS, a dropped connection, and the final
pass over a finished pod. The resume then deferred and replayed the log
again, never getting past the first silent stretch.

Reconnects now send the request read_namespaced_pod_log sends, through
ApiClient.call_api, with sinceTime added. That includes the Accept
header the generated method picks (application/json): the API server
answers a log request that accepts only text/plain with 406.

The test fake rejects the keywords the real client rejects, and a new
test drives the real CoreV1Api and ApiClient down to the urllib3 request
and compares it with the generated method's. Checked live against
Kubernetes 1.37 (kind): a pod writing in bursts, followed with a two
second idle timeout, was read through one reconnect per silence, every
line once and in order.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mestamp

PodLogStream dropped every line whose timestamp was not later than the
last one delivered, on every connection, including the first. CRI-O's
conmon stamps all the lines of one read from the container with the
same time, so on CRI-O (OpenShift among others) a resumed read lost
most events and corrupted the artifacts archive; containerd stamps
each line, where equal stamps are rare but possible.

Lines are now always delivered. The stream counts how many delivered
lines carry the last timestamp, and after a reconnect, which starts at
that second, skips the lines before that timestamp and that many at
it, and nothing else.

A followed connection that ends inside a line no longer delivers the
fragment and moves its cursor past it; the reconnect serves the whole
line. Only a read that does not follow, of a complete log, may end with
a line that lacks its newline.

Timestamps are read as instants, so a runtime that writes an offset
rather than Z, on a node whose clock is not in UTC, is understood, and
sinceTime is sent in UTC.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The final pass over a pod's log, which serves what was written after the
last line read, turned a read error into the end of the log. The resume
then took the job's status from the pod's exit code and released the
pod, with the rest of the events and the artifacts archive unread.

Such an error now raises PodLogIncomplete, which is deliberately not an
OSError: ansible-runner's processor takes an OSError from its input for
the end of the output. The resume fails the read and tries again later,
with the pod still there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A node marks itself unavailable as its dispatcher stops
(inform_cluster_of_shutdown), before the dispatcher's tasks detach from
their jobs, and is ready again only once the new dispatcher has started.
A detached job's lease expires at once, so a peer sweeping in that
window found the hybrid node that runs the job's unit not ready, claimed
the job, could not read the unit, and failed or re-queued it while the
node's receptor was still running it. A routine dispatcher restart on a
hybrid node in a multi-node cluster could fail its running jobs, or
start a second copy of them.

The sweep and the resume plan now treat that node as present while it
is ready, or unavailable for less than the time its peers give a node
that stops sending heartbeats before declaring it lost (Instance.is_lost).
A graceful shutdown stamps last_seen, so a restarting node keeps its
jobs, and one that does not come back gives them up as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
_record_pod runs at the end of the transmit phase and at every lease
renewal until the pod name of a container group job is on its lease. A
database error there escaped into AWXReceptorJob.run, which closed the
stream, released the work unit (deleting the running pod) and failed
the job, over a write the job does not need to keep running. Lease
renewals already tolerate the same errors.

The error is now logged and the record is left for the next renewal
tick to try again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
worker_started on the lease is what a later reader goes by when the
job's work unit or pod is gone: not started means nothing ran and the
job can be re-queued. The flag is written once, when the reader handles
ansible-runner's 'starting' status, and a failed write was only logged.
The lease then said the job never started for as long as it ran, and a
controller crash that also lost the work would re-queue a job that had
run.

The write is now tried again at every lease renewal until it succeeds,
the way the pod name already is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n hook

A project update takes the lock on its project in its pre-run hook and
releases it in its post-run hook. A run that ends without that hook (a
re-queue on shutdown, a lease taken over by another task, ownership
found lost just before finishing) kept the lock's descriptor open in the
dispatcher worker, which lives on after the task. Every later update of
the project on that node, and every job copying the project, then waited
on a lock nobody would release.

These exits now release the lock. A project update whose lease is taken
over also has its work stopped by the task losing it, before the lock
goes: its new owner never reads a project update, it stops and re-runs
it, so the work must not run on without the lock in between.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A task that detached from a job on shutdown kept the job's private data
directory, credentials included, for the periodic cleanup to remove once
the job was done. Only local work needs it: its execution environment
has the directory bind-mounted. Remote work, on an execution node or in
a container group pod, took a copy with its payload. On Kubernetes there
is no periodic cleanup of these directories at all, so after an in-place
dispatcher restart they stayed until the task pod was replaced.

The directory is now kept only for local work, and otherwise removed
when the task ends, as it would have been at the end of the job.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…read

A resume that fails a job whose output is gone, or carries out a cancel
that found nobody reading, finishes the job through finish_run, which
ran the job type's post-run hook. For a project update that hook
deletes the project's stage directory when the update did not succeed,
and this resume does not hold the project's lock: another update of the
same project, writing that directory under the lock, lost it.

These finalizations now skip the post-run hook. They read no output, so
no hook has anything to process; a stale stage directory is removed by
the next update's pre-run hook, under the lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A reader checked that it still owned its job, then removed the lease and
released the work unit or pod, in separate steps; a sweep could claim
the job in between. The reader then removed the claimer's lease and
released the work it was about to read, the claimer's resume found no
lease and stood down, and the reader's own final write was fenced out:
the job stayed running with nobody reading it. A claim, in turn, did
not check that the lease it read as expired still was, so it could take
a job whose reader had just renewed or retired the lease. Expiring a
lease on detach and putting it off were not fenced at all, so a reader
that had lost its job could still expire or defer the claimer's lease.

Now:
- claim_job takes the lease row's lock and checks under it that the
  lease is still expired before it moves the job's task id;
- JobMonitorLease.objects.retire takes the same lock and checks that
  its task still owns the job before it removes the lease, and the
  launch and the resume release the unit or pod only when it succeeded;
- release and defer only touch the lease while the caller owns the job,
  including the pool's cleanup of a dead worker, which no longer reaps a
  job that another task has claimed.

Tests run a claim and a retirement against each other in two
transactions, and fail when either side does not take the lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
requeue_job reset any running job to pending, whoever owned it. A
resume that had lost its job to a claim could still reset a job that
another task was reading. It also removed the lease even when the reset
matched nothing, which, behind an owner check, would have removed the
claimer's lease.

The reset now matches only while the caller's task id is the job's, and
the lease and the abandoned attempt's events go in the same transaction
as the reset, only when it matched; the job is announced and the task
manager scheduled after the commit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A claim gives the new owner one lease (60 seconds) to get its stream
open, and nothing renewed the lease until the stream's own loop did.
Planning (finding the pod, asking receptor) and building the host map
of a large inventory can take longer than that. The next sweep then
claimed the job from a task that was about to read it; when setup always
takes longer than a lease, no task ever keeps the job. A task that lost
the job while planning also went on to act on its plan.

A LeaseKeeper thread now renews the lease while the resume plans and
prepares, and stops before the plan is acted on; a task whose renewal
found the job claimed by another stands down without doing anything
about its plan.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On a shutdown, a job whose execution environment had not received its
payload, and any project update, is cancelled and re-queued. A failed
cancel was only logged: the job was re-queued anyway, its work unit id
cleared and its lease removed, while the original work could still go
on, so the next attempt could run alongside it.

When the cancel fails the job is now left for the next monitor, as on a
detach. The resume re-queues it only once its own cleanup has stopped
what is left, and puts it off while it cannot. PodLogStream.cancel now
reports a pod it could not delete, as a receptor cancel does by raising.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The sweep took the JOB_MONITOR_SWEEP_BATCH oldest expired leases, then
skipped the ones this node may not take: another node's jobs while it
has no control capacity to spare. A node behind a full batch of such
leases never reached its own expired leases, although it takes those
back whatever its capacity, and its own jobs went unread until someone
else took the older ones.

The batch now puts this node's own jobs (the ones it controls, or whose
local unit it runs) first, then the rest by expiry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A resume that finds the job's pod still pending puts the job off, and
every such wait counted against JOB_MONITOR_MAX_DEFERRALS: 120 waits of
30 seconds, an hour at the defaults. ASCENDER_CONTAINER_GROUP_POD_PENDING_TIMEOUT
defaults to two hours, so a pod that is legitimately pending (waiting for
a node, or for quota) failed after an hour, with a reason about
unreachable output rather than the pending timeout.

A WAIT can now say it does not count, and waiting for a pending pod does
not: the pending timeout, which plan_resume already enforces, ends it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A resumed read skips the events that are already saved. They were
tracked as the contiguous prefix from counter 1 plus a set of the saved
counters past the first gap, capped at JOB_EVENT_WORKERS * 1000, while
the event count was seeded with all of them. Past the cap, saved events
were replayed and counted a second time, so emitted_events ended up
higher than the events saved and event_processing_finished stayed false;
each such duplicate also sent the receiver's whole batch into one-by-one
inserts. A gap that large is left by an event the receiver dropped after
its retries, with the rest of a long job saved after it. The prefix, the
count and the set also came from three queries, so an event saved in
between by a receiver that was behind could be skipped without being
counted.

The saved counters are now read as (first, last) ranges, with their
count, in one gaps-and-islands query, and a replayed counter is looked
up by bisection. Memory grows with the number of gaps, not of events,
and the count and what is skipped describe the same moment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The post-run hooks find a job's artifacts through AWX_PRIVATE_DATA_DIR
in the job's stored environment. A resumed read only points that at its
own private data dir when an environment is stored, or when it replays
the worker's 'starting' status. A launch that died before storing one,
for a job whose log start was rotated away before it was read again,
left neither: the job was finished as successful while its inventory
import, or its fact cache, was skipped for want of the directory.

The hooks now take the directory from the task, which knows the resumed
read's own, and fall back to the stored environment for a launch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The kubelet ends a followed log when the container exits, a moment
before the pod's status says the container has terminated. The stream
saw a live pod and reopened the log, which ended at once, and so on, as
fast as the API answered (a pod read and a log read per round), until
the status caught up: dozens of requests at the end of every followed
pod log, seen against a kind cluster.

A followed connection that ended on its own, rather than by dropping
or going idle, is now reopened after FOLLOW_REOPEN_PAUSE (a second), or
not at all if the reader is interrupted meanwhile.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI max review requested due to automatic review settings November 10, 2026 22:56

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved critical recovery and cancellation races can rerun partially executed jobs, mix output across attempts, and skip required completion processing.

29 open issues

And 9 more that still need to be addressed.

15 resolved since last review

Review effort: Max

Comment thread ascender/main/tasks/jobs.py Resolved
Comment thread ascender/main/tasks/jobs.py Resolved
Comment thread ascender/main/tasks/monitor.py Resolved
Comment thread ascender/main/tasks/monitor.py Resolved
Comment thread ascender/main/tasks/receptor.py Resolved
Comment thread ascender/main/tasks/receptor.py Resolved
Comment thread ascender/main/tasks/jobs.py Resolved
Comment thread ascender/main/tasks/monitor.py Resolved
Comment thread ascender/main/tasks/receptor.py Resolved
Comment thread ascender/main/tasks/streams.py Resolved
cigamit and others added 7 commits November 10, 2026 17:52
AWXReceptorJob.finished_callback sends no EOF for a stream that was
abandoned (detached, re-queued or taken over), because the job's output
has not ended. The streamer calls it as it exits on the dropped stream,
which happens while _stop_work drains it. Since "Do not re-queue a job
on shutdown when its work could not be cancelled", the re-queue flag was
only set after the drain, so a re-queued attempt sent an EOF: it
announced the job done to the UI and, for job types that wrap up on EOF,
ran the wrapup for an attempt that was given up.

The flag is set before the stream is dropped again, and switched to
detached when the cancel fails.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
requeue_job updated the job row, then deleted the lease, in one
transaction; claim_job locks the lease row, then updates the job. A
re-queue and a claim of the same job could each hold one row and wait
for the other, and PostgreSQL aborted one of them as a deadlock. That
takes a re-queue of a job whose lease expired while a sweep claims it,
which the lease keeper makes rare, but the abort leaves the job to the
next sweep with its work possibly already stopped.

requeue_job now locks the lease row first, as the claim and the
retirement do. A test runs a claim and a re-queue against each other in
two transactions: in the old order PostgreSQL reports the deadlock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
receptor's work adopt (ansible/receptor#1564) allocates the unit on this
node and starts it tracking the remote one, answering "Adopted" once it
is connected and "Adopt Pending" while it is still connecting
(finalizeAdoptRestart). Only "Adopted" was taken, along with an "Already
Adopted" that receptor never sends, so a pending adoption put the job
off for JOB_MONITOR_RETRY_SECONDS and counted against its deferrals,
although the unit could be read at once; the next resume then found
the unit known here and read it.

"Adopt Pending" now counts as adopted, and "Already Adopted" is gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A resume adopts a unit from its execution or hop node only while that
node is ready, and gave the unit up otherwise: a playbook was failed,
and a job that can run again was re-queued. A node that is merely
unavailable (a failed health check, a link down for a while) may still
be running the unit, so the job could be reported failed, or run a
second time, while its work went on.

The plan now waits for such a node, counted against the deferrals,
until it is ready again or its peers would declare it lost
(Instance.is_lost), and gives the unit up only then.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…etup

An exception while a resume got ready (planning, building its private
data dir, configuring the callback) escaped resume() without putting
the job off. The lease lapsed, the next sweep claimed the job and failed
the same way, and so on: a failure that persists, such as a container
group whose credential is broken or a full disk, kept the job running
forever, never counting against JOB_MONITOR_MAX_DEFERRALS. And the path
that fails a job once its deferrals are spent built a private data dir
too, so it could fail the same way.

Getting ready is now guarded: a failure is logged and counted, through
the owner-fenced deferral. Failing or cancelling a job a resume never
read takes no private data dir and leaves the stored environment alone,
and final_run_hook skips the artifacts when there is no directory, so a
job out of deferrals is failed with nothing but database writes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PodLogStream._raw_readline added each chunk to its buffer and searched
the whole buffer for a newline, so a line arriving in many chunks was
copied and searched whole once per chunk: quadratic in its length. The
artifacts archive is a single base64 line, relayed by the kubelet in
pieces of about 16 KiB, and a large fact cache makes it long. Measured
in the dev container: 0.96 s for 10 MiB, 6.3 s for 25 MiB, 37 s for
50 MiB, every time a resume reads a pod log.

The chunks of a line are now gathered in a list, each searched on its
own for the newline, and joined once: 0.01, 0.03 and 0.07 seconds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eaping it

When a peer claims a job whose unit ran in a lost hybrid or control
node's own receptor, it becomes the job's controller while the job's
execution node stays the lost one, and its resume re-queues the job, or
fails it, within moments. The heartbeat reaps a lost node's jobs every
minute on every peer, and reap() spared a leased job only when the lost
node was its controller: in between the claim and the re-queue, it
failed jobs that were about to be run again.

A control or hybrid node executes only work it controls, so reap() now
leaves all of such a node's leased jobs to the sweep. Reaping an
execution or hop node still reaps the leased jobs whose work is on it,
whose controller would otherwise wait for it forever. The task manager's
orphan reaper, for a node whose record is gone, is unchanged: it cannot
tell a deprovisioned control node from a removed execution node.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI max review requested due to automatic review settings November 11, 2026 00:19

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because interacting recovery and scheduling paths carry critical risks to inventory integrity, correct job finalization, and capacity accounting.

No open issues 🎉

29 resolved since last review

And 9 more resolved.

Review effort: Max

@cigamit
cigamit requested a review from TheWitness November 11, 2026 03:30
cigamit and others added 3 commits November 10, 2026 22:17
A job or inventory update that needs its project synced runs the sync
inline, inside its own task, and goes on once RunProjectUpdate.run()
returns: run() raises when the sync fails. A sync whose lease another
task claimed, after a stall of its reader, is different: the reader
stops the sync's work and returns without finishing it, and the task
that claimed it fails it. The job that started the sync took that return
for success, recorded the sync's revision and ran its playbook against a
half-updated project, without the roles and collections the sync was
installing.

sync_and_copy_without_lock now checks that the sync really ended
successful, and fails the job as for any failed sync otherwise.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A task that stops reading a job without finishing it kept the job's
private data dir whenever the work was local, since the execution
environment of local work that goes on has the directory bind-mounted.
A project update only gets there by losing its lease, and its work is
stopped as it loses it, so nothing uses the directory any more; it was
kept all the same, SCM credentials included, until the periodic cleanup,
which Kubernetes does not run, removed it.

The directory is now kept only for work that goes on: local work of a
job that can be left for the next monitor to read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The sweep took each job from its lease, through which django-polymorphic
gives a plain UnifiedJob, not the Job or InventoryUpdate it is. Such an
object says it is never a container group task and logs as
'unified_job N', so the sweep's warning that it is resuming a job named
the job wrongly. Its decisions were right only because a container group
job's execution node is never a control or hybrid node.

The sweep now loads the batch's jobs through the polymorphic manager,
one query per job type, and the leases no longer bring the job rows
along.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI max review requested due to automatic review settings November 11, 2026 04:37

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Critical concurrency, credential-handling, and event-integrity risks spanning scheduling and recovery paths require human assessment.

No open issues 🎉

Review effort: Max

…amer

process_stream arms signal_state.raise_exception for the wait on the
streamer, so that a SIGTERM interrupts it, and disarms it once the wait
is over. _await_processor disarms it around its database work and then
re-armed it unconditionally, so it returned with it armed whatever its
caller had set. process_stream disarms it right after, so a dispatcher
task was not affected; a test that calls _await_processor directly left
it armed for the rest of its worker, and test_signals.py failed in CI
with SignalExit when it ran later in the same xdist worker.

_await_processor now restores the arming it found on every way out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI max review requested due to automatic review settings November 11, 2026 05:19

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed to evaluate interacting recovery risks, particularly crash-driven resumes that can repeat indefinitely without reaching a failure limit.

No open issues 🎉

Review effort: Max

@cigamit
cigamit merged commit 9938e70 into main Nov 11, 2026
15 checks passed
@cigamit
cigamit deleted the job-monitor-lease branch November 12, 2026 18:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Development

Successfully merging this pull request may close these issues.

4 participants