Repository navigation
Keep running jobs alive across dispatcher and task-pod restarts (job monitor lease) #1198
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
30 commits
Select commit
Hold shift + click to select a range
abfdb16
Keep running jobs alive across dispatcher and task-pod restarts (job …
cigamit e7f0dbc
Send sinceTime on pod log reconnects through a raw API request
cigamit 4469e76
Skip only what a pod log reconnect repeats, not lines that share a ti…
cigamit f14462a
Treat a pod log read that breaks off in its final pass as an error
cigamit 2c7bfe2
Leave a local unit to its node while that node restarts its dispatcher
cigamit 1c28bb8
Keep reading a job when recording its pod name hits a database error
cigamit 84fcc59
Try again to record that the worker started when the write fails
cigamit 2e165ff
Release a project update's lock when its run ends without the post-ru…
cigamit d860e0e
Remove a detached job's private data dir unless its work is local
cigamit eb10d99
Skip the post-run hook when a resume fails or cancels a job it never …
cigamit 56bf1cf
Serialize lease claims and retirements on the lease row
cigamit 1f90860
Re-queue a job only for the task that owns it, in one transaction
cigamit efe6af1
Keep the monitor lease renewed while a resume gets ready to read
cigamit 9e03603
Do not re-queue a job on shutdown when its work could not be cancelled
cigamit 91f24e2
Sweep a node's own expired leases first
cigamit 6d67027
Do not count waiting for a pending pod against the deferral budget
cigamit 77fee30
Track a resumed job's saved events as ranges read in one query
cigamit 608429c
Let the post-run hooks of a resumed job find its artifacts directly
cigamit 43aeed1
Pause before reopening a followed pod log that the kubelet ended
cigamit cf1b63e
Mark a re-queued attempt before its stream drains, so it sends no EOF
cigamit 8396751
Lock the lease before the job when re-queueing it
cigamit cb137c6
Take receptor's "Adopt Pending" answer for an adopted unit
cigamit e6bd265
Wait for an execution node that is unavailable but not yet lost
cigamit e60b137
Count a resume that fails to get ready, and fail a job without that s…
cigamit 5e9e8d6
Read a pod log line that spans many chunks in linear time
cigamit 9a50990
Leave a lost control or hybrid node's leased jobs to the sweep when r…
cigamit a5fc7b3
Fail a job whose inline project sync was taken over, not run it
cigamit 1624944
Remove the private data dir of a project update that lost its lease
cigamit b6df41d
Have the job monitor sweep see each job as its own type
cigamit a3aa821
Leave signal handling as the caller set it after waiting for the stre…
cigamit File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,95 @@ | ||
| # Generated by Django 6.1.1 on 2026-11-10 09:23, then edited: the constraints were split from | ||
| # the job monitor lease model (0233) and preceded by a de-duplication. | ||
|
|
||
| from django.db import migrations, models | ||
|
|
||
| # (table, column of the job reference) | ||
| EVENT_TABLES = ( | ||
| ('main_jobevent', 'job_id'), | ||
| ('main_projectupdateevent', 'project_update_id'), | ||
| ('main_adhoccommandevent', 'ad_hoc_command_id'), | ||
| ('main_inventoryupdateevent', 'inventory_update_id'), | ||
| ('main_systemjobevent', 'system_job_id'), | ||
| ) | ||
|
|
||
|
|
||
| def dedup_events(apps, schema_editor): | ||
| """ | ||
| Remove duplicate (job, job_created, counter) rows before the unique constraints are added. | ||
|
|
||
| ansible-runner numbers the events of a run one by one, so a database should hold no | ||
| duplicates, and this deletes nothing; it is here so that the constraints apply even to one | ||
| that somehow does, keeping the row with the lowest id, the earliest insert. Each table is | ||
| first checked with a scan of its existing (job, job_created, counter) index, which is cheap | ||
| next to the self-join that removes duplicates when there are any. | ||
|
|
||
| NOTE: on a large deployment the event tables are partitioned by job_created with many | ||
| partitions, and ADD CONSTRAINT builds a unique index on every one of them under a lock. | ||
| Build the indexes CONCURRENTLY per partition beforehand if that lock is a concern. | ||
| """ | ||
| with schema_editor.connection.cursor() as cursor: | ||
| for table, job_column in EVENT_TABLES: | ||
| cursor.execute(f'SELECT 1 FROM {table} GROUP BY {job_column}, job_created, counter HAVING count(*) > 1 LIMIT 1') | ||
| if cursor.fetchone() is None: | ||
| continue | ||
| cursor.execute( | ||
| f""" | ||
| DELETE FROM {table} a | ||
| USING {table} b | ||
| WHERE a.id > b.id | ||
| AND a.{job_column} = b.{job_column} | ||
| AND a.job_created = b.job_created | ||
| AND a.counter = b.counter | ||
| """ | ||
| ) | ||
|
|
||
|
|
||
| class Migration(migrations.Migration): | ||
| dependencies = [ | ||
| ('main', '0231_workload_identity_key'), | ||
| ] | ||
|
|
||
| operations = [ | ||
| migrations.RunPython(dedup_events, migrations.RunPython.noop), | ||
| # Each unique index replaces the plain index on the same columns, in the same order | ||
| migrations.AddConstraint( | ||
| model_name='adhoccommandevent', | ||
| constraint=models.UniqueConstraint(fields=('ad_hoc_command', 'job_created', 'counter'), name='main_adhoccommandevent_counter_unique'), | ||
| ), | ||
| migrations.RemoveIndex( | ||
| model_name='adhoccommandevent', | ||
| name='main_adhocc_ad_hoc__a57777_idx', | ||
| ), | ||
| migrations.AddConstraint( | ||
| model_name='inventoryupdateevent', | ||
| constraint=models.UniqueConstraint(fields=('inventory_update', 'job_created', 'counter'), name='main_inventoryupdateevent_counter_unique'), | ||
| ), | ||
| migrations.RemoveIndex( | ||
| model_name='inventoryupdateevent', | ||
| name='main_invent_invento_364dcb_idx', | ||
| ), | ||
| migrations.AddConstraint( | ||
| model_name='jobevent', | ||
| constraint=models.UniqueConstraint(fields=('job', 'job_created', 'counter'), name='main_jobevent_counter_unique'), | ||
| ), | ||
| migrations.RemoveIndex( | ||
| model_name='jobevent', | ||
| name='main_jobeve_job_id_51c382_idx', | ||
| ), | ||
| migrations.AddConstraint( | ||
| model_name='projectupdateevent', | ||
| constraint=models.UniqueConstraint(fields=('project_update', 'job_created', 'counter'), name='main_projectupdateevent_counter_unique'), | ||
| ), | ||
| migrations.RemoveIndex( | ||
| model_name='projectupdateevent', | ||
| name='main_projec_project_69559a_idx', | ||
| ), | ||
| migrations.AddConstraint( | ||
| model_name='systemjobevent', | ||
| constraint=models.UniqueConstraint(fields=('system_job', 'job_created', 'counter'), name='main_systemjobevent_counter_unique'), | ||
| ), | ||
| migrations.RemoveIndex( | ||
| model_name='systemjobevent', | ||
| name='main_system_system__73537a_idx', | ||
| ), | ||
| ] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,37 @@ | ||
| # Generated by Django 6.1.1 on 2026-11-10 09:23 | ||
|
|
||
| import django.db.models.deletion | ||
| from django.db import migrations, models | ||
|
|
||
|
|
||
| class Migration(migrations.Migration): | ||
| dependencies = [ | ||
| ('main', '0232_event_counter_unique'), | ||
| ] | ||
|
|
||
| operations = [ | ||
| migrations.CreateModel( | ||
| name='JobMonitorLease', | ||
| fields=[ | ||
| ( | ||
| 'unified_job', | ||
| models.OneToOneField( | ||
| editable=False, | ||
| on_delete=django.db.models.deletion.CASCADE, | ||
| primary_key=True, | ||
| related_name='monitor_lease', | ||
| serialize=False, | ||
| to='main.unifiedjob', | ||
| ), | ||
| ), | ||
| ('expires', models.DateTimeField(db_index=True, editable=False)), | ||
| ('pod_name', models.CharField(blank=True, default='', editable=False, max_length=253)), | ||
| ('pod_namespace', models.CharField(blank=True, default='', editable=False, max_length=253)), | ||
| ('worker_started', models.BooleanField(default=False, editable=False)), | ||
| ('resumed', models.PositiveIntegerField(default=0, editable=False)), | ||
| ('deferrals', models.PositiveIntegerField(default=0, editable=False)), | ||
| ('created', models.DateTimeField(auto_now_add=True)), | ||
| ('modified', models.DateTimeField(auto_now=True)), | ||
| ], | ||
| ), | ||
| ] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.