Repository navigation
Keep running jobs alive across dispatcher and task-pod restarts (job monitor lease) - #1198
Conversation
eafce0f to
abfdb16
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved critical lease-ownership and recovery flaws can rerun partially executed jobs, lose output, and leave previous executions running after finalization.
22 open issues
Concurrent duplicate stats events double-increment host metrics · New Stale readers can delete a replacement task’s lease and output · New Claims leave execution_node stale and expose local jobs to reaping · New Capped replay double-counts persisted event counters · New Persisted stats events can skip required wrapup reconciliation · New Stale callbacks can consume counters from a new attempt · New Ownership-loss exits skip project lock cleanup · New Detached remote jobs leave credential-bearing directories behind · New Recovery finalization can delete another update’s stage directory · New Lease can expire during setup before its first renewal · New Resume can reset another task’s job and drop its lease · New Lease claim can overwrite a renewed or retired lease · New Retry exhaustion can fail jobs while their work keeps running · New Graceful shutdown can make running local work appear lost · New Cancellation is signaled before its flag commits · New Unobserved worker start can cause unsafe job retries · New Failed cancellation can trigger a retry while old work continues · New Unsupported since_time parameter breaks Kubernetes log recovery · New Final-pass read failures are mistaken for normal EOF · New Equal-timestamp log lines are dropped on uninterrupted reads · New
And 2 more that still need to be addressed.
What changed in this PR
Introduces lease-based recovery in Ascender’s task engine so running automation can continue while another dispatcher resumes monitoring after a restart.
Changes:
- Adds monitor leases, recovery planning, shutdown detachment, and ownership-fenced completion.
- Resumes output from Kubernetes pods or receptor units, with replay deduplication and updated reaper behavior.
- Adds migrations, recovery tests, configurable settings, and operator documentation.
| File | Description |
|---|---|
docs/job_monitor_lease.md |
Documents recovery behavior, limitations, and upgrades. |
ascender/settings/typed.py |
Adds types for monitor settings. |
ascender/settings/defaults.py |
Defines defaults and schedules the monitor sweep. |
ascender/main/utils/update_model.py |
Adds ownership fencing and preserves locking during retries. |
ascender/main/tests/unit/tasks/test_streams.py |
Tests pod and receptor stream behavior. |
ascender/main/tests/functional/test_job_monitor.py |
Tests leases, claims, recovery, and reapers. |
ascender/main/tests/functional/api/test_unified_jobs_stdout.py |
Gives stdout fixtures distinct event counters. |
ascender/main/tasks/streams.py |
Adds reconnectable Kubernetes pod-log streaming. |
ascender/main/tasks/receptor.py |
Adds lease renewal, unit adoption, and shutdown handling. |
ascender/main/tasks/monitor.py |
Implements sweeping, claiming, recovery planning, and re-queueing. |
ascender/main/tasks/jobs.py |
Adds resume handling and shared, fenced finalization. |
ascender/main/tasks/callback.py |
Adds replay filtering and preserves masked environments. |
ascender/main/tasks/__init__.py |
Registers monitor tasks. |
ascender/main/scheduler/task_manager.py |
Adjusts orphan reaping for leased jobs. |
ascender/main/models/monitor.py |
Defines the lease model and manager. |
ascender/main/models/events.py |
Adds event uniqueness and replay-aware host summaries. |
ascender/main/models/__init__.py |
Exports lease-related classes. |
ascender/main/migrations/0233_job_monitor_lease.py |
Creates the monitor lease table. |
ascender/main/migrations/0232_event_counter_unique.py |
Deduplicates events and replaces indexes with unique constraints. |
ascender/main/dispatch/worker/callback.py |
Drops duplicate events and guards stats notifications. |
ascender/main/dispatch/reaper.py |
Leaves leased controller jobs to monitor recovery. |
ascender/main/dispatch/pool.py |
Releases dead workers’ leases for recovery. |
Review effort: Max
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved critical lease-ownership races and replay-handling defects can strand jobs, duplicate execution, and lose events or completion notifications.
32 open issues
A duplicate stats row is not proof that its notification wrapup finished. If the original receiver… · New If a reader receives a shutdown signal after its expired lease has been claimed, its detach path… · New Callback receivers can insert events betweenabove.count()and the skip-set query, so a newly… · New If the launch saved nojob_envand log rotation removedstarting, this guard never exposes the… · New Deleting saved events twice does not isolate a re-queued attempt from its predecessor. Events… · Newworker_started=Falseonly means the controller did not recordstarting; the worker may already… · New A sweep can claim an expired lease after_still_owns_job()returns but before lease deletion, so… · New If a followed connection ends halfway through a record, this returns the fragment and_fill()… · New Distinct pod log records can share a timestamp, but this comparison drops every equal-stamped… · New Equal-timestamp log lines are dropped on uninterrupted reads Final-pass read failures are mistaken for normal EOF Unsupported since_time parameter breaks Kubernetes log recovery Failed cancellation can trigger a retry while old work continues Unobserved worker start can cause unsafe job retries Cancellation is signaled before its flag commits Graceful shutdown can make running local work appear lost Retry exhaustion can fail jobs while their work keeps running Lease claim can overwrite a renewed or retired lease Resume can reset another task’s job and drop its lease Lease can expire during setup before its first renewal
And 12 more that still need to be addressed.
Review effort: Max
|
This addresses imho one of the biggest weaknesses of AWX - very much looking forward to this! 😍 |
The generated kubernetes client (36.0.3) leaves sinceTime out of read_namespaced_pod_log and rejects since_time as an unexpected keyword argument, so every reconnect of a pod log after the first line raised ApiTypeError: a followed log that stayed silent for JOB_MONITOR_STREAM_IDLE_SECONDS, a dropped connection, and the final pass over a finished pod. The resume then deferred and replayed the log again, never getting past the first silent stretch. Reconnects now send the request read_namespaced_pod_log sends, through ApiClient.call_api, with sinceTime added. That includes the Accept header the generated method picks (application/json): the API server answers a log request that accepts only text/plain with 406. The test fake rejects the keywords the real client rejects, and a new test drives the real CoreV1Api and ApiClient down to the urllib3 request and compares it with the generated method's. Checked live against Kubernetes 1.37 (kind): a pod writing in bursts, followed with a two second idle timeout, was read through one reconnect per silence, every line once and in order. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mestamp PodLogStream dropped every line whose timestamp was not later than the last one delivered, on every connection, including the first. CRI-O's conmon stamps all the lines of one read from the container with the same time, so on CRI-O (OpenShift among others) a resumed read lost most events and corrupted the artifacts archive; containerd stamps each line, where equal stamps are rare but possible. Lines are now always delivered. The stream counts how many delivered lines carry the last timestamp, and after a reconnect, which starts at that second, skips the lines before that timestamp and that many at it, and nothing else. A followed connection that ends inside a line no longer delivers the fragment and moves its cursor past it; the reconnect serves the whole line. Only a read that does not follow, of a complete log, may end with a line that lacks its newline. Timestamps are read as instants, so a runtime that writes an offset rather than Z, on a node whose clock is not in UTC, is understood, and sinceTime is sent in UTC. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The final pass over a pod's log, which serves what was written after the last line read, turned a read error into the end of the log. The resume then took the job's status from the pod's exit code and released the pod, with the rest of the events and the artifacts archive unread. Such an error now raises PodLogIncomplete, which is deliberately not an OSError: ansible-runner's processor takes an OSError from its input for the end of the output. The resume fails the read and tries again later, with the pod still there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A node marks itself unavailable as its dispatcher stops (inform_cluster_of_shutdown), before the dispatcher's tasks detach from their jobs, and is ready again only once the new dispatcher has started. A detached job's lease expires at once, so a peer sweeping in that window found the hybrid node that runs the job's unit not ready, claimed the job, could not read the unit, and failed or re-queued it while the node's receptor was still running it. A routine dispatcher restart on a hybrid node in a multi-node cluster could fail its running jobs, or start a second copy of them. The sweep and the resume plan now treat that node as present while it is ready, or unavailable for less than the time its peers give a node that stops sending heartbeats before declaring it lost (Instance.is_lost). A graceful shutdown stamps last_seen, so a restarting node keeps its jobs, and one that does not come back gives them up as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
_record_pod runs at the end of the transmit phase and at every lease renewal until the pod name of a container group job is on its lease. A database error there escaped into AWXReceptorJob.run, which closed the stream, released the work unit (deleting the running pod) and failed the job, over a write the job does not need to keep running. Lease renewals already tolerate the same errors. The error is now logged and the record is left for the next renewal tick to try again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
worker_started on the lease is what a later reader goes by when the job's work unit or pod is gone: not started means nothing ran and the job can be re-queued. The flag is written once, when the reader handles ansible-runner's 'starting' status, and a failed write was only logged. The lease then said the job never started for as long as it ran, and a controller crash that also lost the work would re-queue a job that had run. The write is now tried again at every lease renewal until it succeeds, the way the pod name already is. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n hook A project update takes the lock on its project in its pre-run hook and releases it in its post-run hook. A run that ends without that hook (a re-queue on shutdown, a lease taken over by another task, ownership found lost just before finishing) kept the lock's descriptor open in the dispatcher worker, which lives on after the task. Every later update of the project on that node, and every job copying the project, then waited on a lock nobody would release. These exits now release the lock. A project update whose lease is taken over also has its work stopped by the task losing it, before the lock goes: its new owner never reads a project update, it stops and re-runs it, so the work must not run on without the lock in between. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A task that detached from a job on shutdown kept the job's private data directory, credentials included, for the periodic cleanup to remove once the job was done. Only local work needs it: its execution environment has the directory bind-mounted. Remote work, on an execution node or in a container group pod, took a copy with its payload. On Kubernetes there is no periodic cleanup of these directories at all, so after an in-place dispatcher restart they stayed until the task pod was replaced. The directory is now kept only for local work, and otherwise removed when the task ends, as it would have been at the end of the job. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…read A resume that fails a job whose output is gone, or carries out a cancel that found nobody reading, finishes the job through finish_run, which ran the job type's post-run hook. For a project update that hook deletes the project's stage directory when the update did not succeed, and this resume does not hold the project's lock: another update of the same project, writing that directory under the lock, lost it. These finalizations now skip the post-run hook. They read no output, so no hook has anything to process; a stale stage directory is removed by the next update's pre-run hook, under the lock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A reader checked that it still owned its job, then removed the lease and released the work unit or pod, in separate steps; a sweep could claim the job in between. The reader then removed the claimer's lease and released the work it was about to read, the claimer's resume found no lease and stood down, and the reader's own final write was fenced out: the job stayed running with nobody reading it. A claim, in turn, did not check that the lease it read as expired still was, so it could take a job whose reader had just renewed or retired the lease. Expiring a lease on detach and putting it off were not fenced at all, so a reader that had lost its job could still expire or defer the claimer's lease. Now: - claim_job takes the lease row's lock and checks under it that the lease is still expired before it moves the job's task id; - JobMonitorLease.objects.retire takes the same lock and checks that its task still owns the job before it removes the lease, and the launch and the resume release the unit or pod only when it succeeded; - release and defer only touch the lease while the caller owns the job, including the pool's cleanup of a dead worker, which no longer reaps a job that another task has claimed. Tests run a claim and a retirement against each other in two transactions, and fail when either side does not take the lock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
requeue_job reset any running job to pending, whoever owned it. A resume that had lost its job to a claim could still reset a job that another task was reading. It also removed the lease even when the reset matched nothing, which, behind an owner check, would have removed the claimer's lease. The reset now matches only while the caller's task id is the job's, and the lease and the abandoned attempt's events go in the same transaction as the reset, only when it matched; the job is announced and the task manager scheduled after the commit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A claim gives the new owner one lease (60 seconds) to get its stream open, and nothing renewed the lease until the stream's own loop did. Planning (finding the pod, asking receptor) and building the host map of a large inventory can take longer than that. The next sweep then claimed the job from a task that was about to read it; when setup always takes longer than a lease, no task ever keeps the job. A task that lost the job while planning also went on to act on its plan. A LeaseKeeper thread now renews the lease while the resume plans and prepares, and stops before the plan is acted on; a task whose renewal found the job claimed by another stands down without doing anything about its plan. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On a shutdown, a job whose execution environment had not received its payload, and any project update, is cancelled and re-queued. A failed cancel was only logged: the job was re-queued anyway, its work unit id cleared and its lease removed, while the original work could still go on, so the next attempt could run alongside it. When the cancel fails the job is now left for the next monitor, as on a detach. The resume re-queues it only once its own cleanup has stopped what is left, and puts it off while it cannot. PodLogStream.cancel now reports a pod it could not delete, as a receptor cancel does by raising. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The sweep took the JOB_MONITOR_SWEEP_BATCH oldest expired leases, then skipped the ones this node may not take: another node's jobs while it has no control capacity to spare. A node behind a full batch of such leases never reached its own expired leases, although it takes those back whatever its capacity, and its own jobs went unread until someone else took the older ones. The batch now puts this node's own jobs (the ones it controls, or whose local unit it runs) first, then the rest by expiry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A resume that finds the job's pod still pending puts the job off, and every such wait counted against JOB_MONITOR_MAX_DEFERRALS: 120 waits of 30 seconds, an hour at the defaults. ASCENDER_CONTAINER_GROUP_POD_PENDING_TIMEOUT defaults to two hours, so a pod that is legitimately pending (waiting for a node, or for quota) failed after an hour, with a reason about unreachable output rather than the pending timeout. A WAIT can now say it does not count, and waiting for a pending pod does not: the pending timeout, which plan_resume already enforces, ends it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A resumed read skips the events that are already saved. They were tracked as the contiguous prefix from counter 1 plus a set of the saved counters past the first gap, capped at JOB_EVENT_WORKERS * 1000, while the event count was seeded with all of them. Past the cap, saved events were replayed and counted a second time, so emitted_events ended up higher than the events saved and event_processing_finished stayed false; each such duplicate also sent the receiver's whole batch into one-by-one inserts. A gap that large is left by an event the receiver dropped after its retries, with the rest of a long job saved after it. The prefix, the count and the set also came from three queries, so an event saved in between by a receiver that was behind could be skipped without being counted. The saved counters are now read as (first, last) ranges, with their count, in one gaps-and-islands query, and a replayed counter is looked up by bisection. Memory grows with the number of gaps, not of events, and the count and what is skipped describe the same moment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The post-run hooks find a job's artifacts through AWX_PRIVATE_DATA_DIR in the job's stored environment. A resumed read only points that at its own private data dir when an environment is stored, or when it replays the worker's 'starting' status. A launch that died before storing one, for a job whose log start was rotated away before it was read again, left neither: the job was finished as successful while its inventory import, or its fact cache, was skipped for want of the directory. The hooks now take the directory from the task, which knows the resumed read's own, and fall back to the stored environment for a launch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The kubelet ends a followed log when the container exits, a moment before the pod's status says the container has terminated. The stream saw a live pod and reopened the log, which ended at once, and so on, as fast as the API answered (a pod read and a log read per round), until the status caught up: dozens of requests at the end of every followed pod log, seen against a kind cluster. A followed connection that ended on its own, rather than by dropping or going idle, is now reopened after FOLLOW_REOPEN_PAUSE (a second), or not at all if the reader is interrupted meanwhile. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved critical recovery and cancellation races can rerun partially executed jobs, mix output across attempts, and skip required completion processing.
29 open issues
Keep project protection until the old unit has stopped · New Guard destructive cleanup against concurrent lease claims · New Preserve local-holder classification when claiming lost work · New Wait for unavailable remote nodes that are not yet lost · New Do not requeue while remote cancellation is pending · New Skip finalization until lease retirement succeeds · New Keep lease renewal active while opening the output stream · New Set abandonment flags before draining the processor · New A sweep can claim an expired lease after_still_owns_job()returns but before lease deletion, so…worker_started=Falseonly means the controller did not recordstarting; the worker may already… Deleting saved events twice does not isolate a re-queued attempt from its predecessor. Events… If the launch saved nojob_envand log rotation removedstarting, this guard never exposes the… A duplicate stats row is not proof that its notification wrapup finished. If the original receiver… Failed cancellation can trigger a retry while old work continues Unobserved worker start can cause unsafe job retries Cancellation is signaled before its flag commits Retry exhaustion can fail jobs while their work keeps running Lease claim can overwrite a renewed or retired lease Resume can reset another task’s job and drop its lease Lease can expire during setup before its first renewal
And 9 more that still need to be addressed.
15 resolved since last review
Distinct pod log records can share a timestamp, but this comparison drops every equal-stamped… If a followed connection ends halfway through a record, this returns the fragment and_fill()… Callback receivers can insert events betweenabove.count()and the skip-set query, so a newly… If a reader receives a shutdown signal after its expired lease has been claimed, its detach path… Equal-timestamp log lines are dropped on uninterrupted reads Final-pass read failures are mistaken for normal EOF Unsupported since_time parameter breaks Kubernetes log recovery Graceful shutdown can make running local work appear lost Recovery finalization can delete another update’s stage directory Detached remote jobs leave credential-bearing directories behind Ownership-loss exits skip project lock cleanup Capped replay double-counts persisted event counters A transient database error while recording the pod name escapes_record_pod(), closes the live… Pending-pod waits incorrectly consume the deferral budget Batch limits can starve this node’s expired leases
Review effort: Max
AWXReceptorJob.finished_callback sends no EOF for a stream that was abandoned (detached, re-queued or taken over), because the job's output has not ended. The streamer calls it as it exits on the dropped stream, which happens while _stop_work drains it. Since "Do not re-queue a job on shutdown when its work could not be cancelled", the re-queue flag was only set after the drain, so a re-queued attempt sent an EOF: it announced the job done to the UI and, for job types that wrap up on EOF, ran the wrapup for an attempt that was given up. The flag is set before the stream is dropped again, and switched to detached when the cancel fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
requeue_job updated the job row, then deleted the lease, in one transaction; claim_job locks the lease row, then updates the job. A re-queue and a claim of the same job could each hold one row and wait for the other, and PostgreSQL aborted one of them as a deadlock. That takes a re-queue of a job whose lease expired while a sweep claims it, which the lease keeper makes rare, but the abort leaves the job to the next sweep with its work possibly already stopped. requeue_job now locks the lease row first, as the claim and the retirement do. A test runs a claim and a re-queue against each other in two transactions: in the old order PostgreSQL reports the deadlock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
receptor's work adopt (ansible/receptor#1564) allocates the unit on this node and starts it tracking the remote one, answering "Adopted" once it is connected and "Adopt Pending" while it is still connecting (finalizeAdoptRestart). Only "Adopted" was taken, along with an "Already Adopted" that receptor never sends, so a pending adoption put the job off for JOB_MONITOR_RETRY_SECONDS and counted against its deferrals, although the unit could be read at once; the next resume then found the unit known here and read it. "Adopt Pending" now counts as adopted, and "Already Adopted" is gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A resume adopts a unit from its execution or hop node only while that node is ready, and gave the unit up otherwise: a playbook was failed, and a job that can run again was re-queued. A node that is merely unavailable (a failed health check, a link down for a while) may still be running the unit, so the job could be reported failed, or run a second time, while its work went on. The plan now waits for such a node, counted against the deferrals, until it is ready again or its peers would declare it lost (Instance.is_lost), and gives the unit up only then. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…etup An exception while a resume got ready (planning, building its private data dir, configuring the callback) escaped resume() without putting the job off. The lease lapsed, the next sweep claimed the job and failed the same way, and so on: a failure that persists, such as a container group whose credential is broken or a full disk, kept the job running forever, never counting against JOB_MONITOR_MAX_DEFERRALS. And the path that fails a job once its deferrals are spent built a private data dir too, so it could fail the same way. Getting ready is now guarded: a failure is logged and counted, through the owner-fenced deferral. Failing or cancelling a job a resume never read takes no private data dir and leaves the stored environment alone, and final_run_hook skips the artifacts when there is no directory, so a job out of deferrals is failed with nothing but database writes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PodLogStream._raw_readline added each chunk to its buffer and searched the whole buffer for a newline, so a line arriving in many chunks was copied and searched whole once per chunk: quadratic in its length. The artifacts archive is a single base64 line, relayed by the kubelet in pieces of about 16 KiB, and a large fact cache makes it long. Measured in the dev container: 0.96 s for 10 MiB, 6.3 s for 25 MiB, 37 s for 50 MiB, every time a resume reads a pod log. The chunks of a line are now gathered in a list, each searched on its own for the newline, and joined once: 0.01, 0.03 and 0.07 seconds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eaping it When a peer claims a job whose unit ran in a lost hybrid or control node's own receptor, it becomes the job's controller while the job's execution node stays the lost one, and its resume re-queues the job, or fails it, within moments. The heartbeat reaps a lost node's jobs every minute on every peer, and reap() spared a leased job only when the lost node was its controller: in between the claim and the re-queue, it failed jobs that were about to be run again. A control or hybrid node executes only work it controls, so reap() now leaves all of such a node's leased jobs to the sweep. Reaping an execution or hop node still reaps the leased jobs whose work is on it, whose controller would otherwise wait for it forever. The task manager's orphan reaper, for a node whose record is gone, is unchanged: it cannot tell a deprovisioned control node from a removed execution node. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
🔵 Needs a closer look
Human assessment is needed because interacting recovery and scheduling paths carry critical risks to inventory integrity, correct job finalization, and capacity accounting.
No open issues 🎉
29 resolved since last review
Set abandonment flags before draining the processor Keep lease renewal active while opening the output stream Skip finalization until lease retirement succeeds Do not requeue while remote cancellation is pending Wait for unavailable remote nodes that are not yet lost Preserve local-holder classification when claiming lost work Guard destructive cleanup against concurrent lease claims Keep project protection until the old unit has stopped A sweep can claim an expired lease after_still_owns_job()returns but before lease deletion, so…worker_started=Falseonly means the controller did not recordstarting; the worker may already… Deleting saved events twice does not isolate a re-queued attempt from its predecessor. Events… If the launch saved nojob_envand log rotation removedstarting, this guard never exposes the… A duplicate stats row is not proof that its notification wrapup finished. If the original receiver… Failed cancellation can trigger a retry while old work continues Unobserved worker start can cause unsafe job retries Cancellation is signaled before its flag commits Retry exhaustion can fail jobs while their work keeps running Lease claim can overwrite a renewed or retired lease Resume can reset another task’s job and drop its lease Lease can expire during setup before its first renewal
And 9 more resolved.
Review effort: Max
A job or inventory update that needs its project synced runs the sync inline, inside its own task, and goes on once RunProjectUpdate.run() returns: run() raises when the sync fails. A sync whose lease another task claimed, after a stall of its reader, is different: the reader stops the sync's work and returns without finishing it, and the task that claimed it fails it. The job that started the sync took that return for success, recorded the sync's revision and ran its playbook against a half-updated project, without the roles and collections the sync was installing. sync_and_copy_without_lock now checks that the sync really ended successful, and fails the job as for any failed sync otherwise. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A task that stops reading a job without finishing it kept the job's private data dir whenever the work was local, since the execution environment of local work that goes on has the directory bind-mounted. A project update only gets there by losing its lease, and its work is stopped as it loses it, so nothing uses the directory any more; it was kept all the same, SCM credentials included, until the periodic cleanup, which Kubernetes does not run, removed it. The directory is now kept only for work that goes on: local work of a job that can be left for the next monitor to read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The sweep took each job from its lease, through which django-polymorphic gives a plain UnifiedJob, not the Job or InventoryUpdate it is. Such an object says it is never a container group task and logs as 'unified_job N', so the sweep's warning that it is resuming a job named the job wrongly. Its decisions were right only because a container group job's execution node is never a control or hybrid node. The sweep now loads the batch's jobs through the polymorphic manager, one query per job type, and the leases no longer bring the job rows along. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…amer process_stream arms signal_state.raise_exception for the wait on the streamer, so that a SIGTERM interrupts it, and disarms it once the wait is over. _await_processor disarms it around its database work and then re-armed it unconditionally, so it returned with it armed whatever its caller had set. process_stream disarms it right after, so a dispatcher task was not affected; a test that calls _await_processor directly left it armed for the rest of its worker, and test_signals.py failed in CI with SignalExit when it ran later in the same xdist worker. _await_processor now restores the arming it found on every way out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>


Summary
Running jobs now survive a restart or crash of the dispatcher that monitors them, including the replacement of a task pod on Kubernetes.
Today a graceful stop cancels the job's receptor work unit and fails the job with "Task was canceled due to receiving a shutdown signal." For a container group, cancelling the unit also deletes the job's pod. A crash ends the same way: the restarted dispatcher, or a peer's heartbeat, reaps the job as failed, and the pod reaper then deletes its pod.
With this change the automation keeps running. Another control node, or the same one once it is back, picks up reading the job's output from where it survived. The job then finishes normally, with its complete output.
This is an alternative to the cross-controller job adoption that upstream AWX is building in ansible/awx#16669 and the stacked drafts #16728 to #16734. It reuses several of their pieces, but it decides who reads a job with a lease on the job rather than from the health of its controller. The comparison is under "How this differs from upstream" below.
Design notes for reviewers and operators are in
docs/job_monitor_lease.md.How it works
flowchart TD A["Job handed to receptor"] --> B["Launching task reads the output and renews the lease every 10 s"] B -->|"output read"| C["Lease removed, then post-run hooks and final status"] B -->|"shutdown, worker started"| D["Detach: the work keeps running and the lease is expired at once"] B -->|"shutdown, worker had no payload yet"| R["Cancel the unit and re-queue the job"] B -->|"dispatcher killed"| E["Lease lapses after 60 s"] D --> S["Sweep on any ready control node, every 30 s: claims the job by compare-and-set on celery_task_id"] E --> S S --> P{"Where did the output survive?"} P -->|"pod log or receptor unit"| Q["Read it from the start, skipping events already saved"] Q --> C P -->|"not reachable yet"| W["Put off 30 s, at most 120 times"] W --> S P -->|"gone, and nothing ran or the job repeats safely"| R P -->|"gone"| F["Fail the job with the reason"]The lease.
JobMonitorLeaseis a new table with one row per job whose output is being read. The task that launched the job creates the row once receptor has the whole payload. It renews the row everyJOB_MONITOR_RENEW_SECONDSwhile it reads, and removes it as soon as the output has been read. That happens before the post-run hooks, which can run longer than a lease lasts. The row also records two facts a later reader needs:starting(worker_started), meaning the execution environment received its payload.Shutdown or cancel.
UnifiedJob.cancel()commits the job'scancel_flagbefore it signals the dispatcher, so the flag tells a user's cancel from a shutdown. On a shutdown, the task reading a job:The sweep.
sweep_job_monitorsruns every 30 seconds on every ready control node. It claims each expired lease of a running job with one compare-and-set on the job'scelery_task_id, so two nodes cannot both take a job. The winner sets itself ascontroller_nodeand dispatchesresume_jobunder the new task id. A node may claim:Work that ran in a control or hybrid node's own receptor can be read only through that node. While that node is ready, or restarting its dispatcher, nobody else claims the job. The sweep also deletes expired leases of jobs that ended while nobody was reading them.
Resuming.
BaseTask.resume()asksplan_resume()where the output survived:PodLogStream). A running pod is followed, reconnecting from the last timestamp; a finished pod is read once.work results(ReceptorUnitStream)work adopt, thenwork results. This needs a receptor newer than 1.6.9; see Limits.Some cases need waiting or cleanup:
ASCENDER_CONTAINER_GROUP_POD_PENDING_TIMEOUT.JOB_MONITOR_WEDGE_SECONDS, never got its payload. It is deleted and the job is re-queued.JOB_MONITOR_RETRY_SECONDS, at mostJOB_MONITOR_MAX_DEFERRALStimes. Waiting for a pending pod does not count toward that limit; the pending timeout bounds it.Replaying without duplicates. The stream goes to ansible-runner's process streamer unchanged, from the beginning, so events are parsed, saved and broadcast exactly as during a launch. Replaying from the start also recovers events that were still queued in a lost control pod's valkey.
RunnerCallbackskips counters that are already saved, read as ranges together with their count in one query.(job, job_created, counter)on all five event tables is the guarantee. It replaces the plain index on the same columns, so inserts maintain no more indexes than before.finishedis taken from the wrapup event, so a job that ended while nobody was watching is not stamped late.Fencing. The final status write uses
update_model(owner_task_id=...). It locks the row and raisesNotOwnerifcelery_task_idhas moved on. A reader whose job was claimed by another node finds out at its next renewal (LeaseLost). It then stops without finalizing or releasing anything. A project update also stops its work, since it never runs without its task's lock on the project, and a job whose inline sync this was fails with it. Claims and retirements serialize on the lease row: a claim re-checks under the row's lock that the lease is still expired, and a reader retires its lease under the same lock before it releases the unit or pod, so neither can act on a job the other has just taken or finished. Expiring and putting off a lease are fenced by the owner's task id as well.Re-queued jobs keep their id and start over from pending. The abandoned attempt's events are deleted when the job is re-queued, and again just before the new attempt is submitted. A job launched with passwords prompted at launch is failed instead, because those passwords are cleared from the job when it starts.
Credentials. A resume never overwrites the
job_envthe launch stored, which has the injected credential values masked. A resume cannot rebuild that masking. If the launch stored no environment, the resume stores one with every value a credential could have set hidden.Reapers.
startup_reaping, the heartbeat'sreap(), the pool's dead-worker cleanup and the task manager's orphan reaper leave a leased job to the sweep when the node in question is the job's controller; the heartbeat'sreap()of a control or hybrid node, which executes only work it controls, leaves all its leased jobs to the sweep. A leased job whose execution or hop node is lost is reaped as before: its controller is still reading it, and receptor would wait for that node forever.How this differs from upstream
Upstream infers that a job has lost its reader from the health of the job's controller, and then decides who adopts it. Here the reader states it directly by renewing a lease on the job. An expired lease is the only trigger, and one compare-and-set on the job row is the only arbiter.
Upstream work as of 2026-11-10 (none of it merged):
AAP-89602: implement cross-controller job adoption ansible/awx#16669 (open; +8,150/−292 in 28 files, about 2,200 lines of it product code). A surviving controller adopts the running jobs of a controller it judges lost from heartbeats. The PR covers:
work adoptfor mesh units and a status-based fence on the final write.It needs receptor #1564, #1605, #1606 and #1607. These are merged to receptor devel but are in no release; 1.6.9 has none of them. It also relies on dispatcherd behaviour such as
on_duplicate='discard', which Ascender's dispatcher does not have.#16728 to #16734 (drafts, stacked on devel rather than on #16669). They make the heartbeat's adoption decisions testable: a pure decision module, a fake receptor and an injectable clock. #16731 adds a second fence, claim-then-dispatch with
owner_task_id/NotOwner. #16733 and #16734 are tests only.owner_task_id/NotOwnerin #16731NotOwner, as in #16731, plusLeaseLostso that a stale reader stops at its next renewal; claims and retirements serialize on the lease rowpod_name, receptor devel only)work adopt(receptor devel)worker_started). Silence is used only for a pod whose launch died beforestarting.deferralscolumn on the lease rowbuild_safe_env, which masks by variable name only, so a custom credential value under an ordinary name is stored in clear textjob_envis kept. Without one, every value a credential could have set is hidden.(job, counter, job_created)constraint, receiver dedup and a skip in the callbackThe following are ported from upstream: replay from the start with counter skipping, the unique constraint, receiver and host-summary dedup,
finishedfrom the wrapup event, the host map rebuilt from inventory data (#16669), and the compare-and-set claim with theNotOwnerfence (#16731). The scenario lists in #16733 and #16734 served as a review checklist.What changes for users and operators
Stopping or restarting a dispatcher or task pod no longer fails running jobs.
The automation runs throughout; only the saving of its events waits.
A resumed job says so in
job_explanation: "Output monitoring was resumed on<node>after the node reading it stopped."A job that had not started running, and any project update, is re-queued instead of failed: "The job was re-queued because …". The existing
pre_startclears that explanation when the job starts again.There is a new periodic task
job_monitor_sweep(30 s), and there are new lifecycle events:monitor_leased,pod_recorded,monitor_resumed,monitor_detached,monitor_requeued,monitor_resume_failedandmonitor_resume_canceled.The API and UI are unchanged. The lease table is internal.
Kubernetes: reading a pod's log needs
getonpods/log. receptor already needs that permission for container groups, so no operator or RBAC change is needed. As a follow-up, the operator's long termination grace period can be shortened once this ships.New settings (all optional)
JOB_MONITOR_LEASE_SECONDSJOB_MONITOR_RENEW_SECONDSJOB_MONITOR_SWEEP_BATCHJOB_MONITOR_RETRY_SECONDSJOB_MONITOR_MAX_DEFERRALSJOB_MONITOR_WEDGE_SECONDSJOB_MONITOR_STREAM_IDLE_SECONDSUpgrade notes
0232_event_counter_uniquechecks each event table for duplicate(job, job_created, counter)rows and deletes the later copies; a healthy database has none. It then adds the unique constraints and drops the plain indexes they replace. On large event tables this costs about as much as reindexing each partition, under a lock on each table.0233_job_monitor_leasecreates the new table.Limits
containerLogMaxSize(10 MiB by default) while unmonitored loses the unsaved events in between. Its final artifacts are always in the current file.work adopt(Add work adopt command to track a remote work unit ansible/receptor#1564, not in 1.6.9). It is covered only by tests with a fake receptor. With a receptor without it, a job whose controller is lost while the job runs on an execution node is retried untilJOB_MONITOR_MAX_DEFERRALSand then failed, about an hour at the defaults. Today such a job fails within minutes.Testing
check_migrationsandcheck_settingsare clean, and so areruff checkandruff format --check.tests/functional/test_job_monitor.py(109 tests): lease fencing, the sweep's claim rules, everyplan_resumeoutcome, resume, shutdown handling per job type, the reapers and re-queueing.tests/unit/tasks/test_streams.py(35 tests): pod log reconnects and timeouts, connection close order, receptor stream stall detection.successful, 36/36 events in ordersuccessful, 36/36, pod removed at the endSIGKILLto the dispatcher process groupsuccessful, 36/36finishedback-dated to the pod's end, 36/36, pod removedsuccessful, 36/36 events in ordersuccessfulwith 43 events in ordercanceledat once, and the lease and pod were gone within a minute.Where to start reviewing
ascender/main/models/monitor.py: the lease model and its manager.ascender/main/tasks/monitor.py: the sweep, the claim,plan_resumeand re-queueing.ascender/main/tasks/receptor.py: lease start and renewal inAWXReceptorJob, shutdown handling (_handle_signal), andReceptorUnitStream, includingwork adopt.ascender/main/tasks/streams.py:PodLogStream.ascender/main/tasks/jobs.py:BaseTask.resume(), the fencedfinish_run, and clearing an abandoned attempt's events before submitting.ascender/main/tasks/callback.py: the replay skip, the host map on resume, andjob_envon resume.ascender/main/dispatch/worker/callback.py,ascender/main/models/events.pyand migration 0232: duplicate handling.ascender/main/utils/update_model.py:NotOwner. Retries now keepselect_for_update, which they used to drop.ascender/main/dispatch/reaper.py,ascender/main/dispatch/pool.pyandascender/main/scheduler/task_manager.py: the reapers.Trying it on the dev environment
pause: seconds=5.docker exec tools_ascender_1 supervisorctl restart ascender-processes:dispatcher, or sendSIGKILLto the dispatcher's process group.running. Its output continues in the UI about a minute after a restart, or a minute and a half after a kill. It endssuccessful, with events numbered from 1 without gaps and the "Output monitoring was resumed" explanation.kubectl get podsas well. The restart does not touch the automation pod, and the pod is removed when the job finishes.