Repository navigation
feat: optionally let jobs run while an ad hoc command uses the inventory - #1208
Open
fernandorocagonzalez wants to merge 1 commit into
Open
fernandorocagonzalez wants to merge 1 commit into
fernandorocagonzalez wants to merge 1 commit into
Conversation
A running ad hoc command keeps every job and every other ad hoc command against the same inventory pending until it finishes. With a long ad hoc command, like a package update across many hosts, that holds back work that has nothing to do with it. The new allow_jobs_while_ad_hoc flag on the inventory lifts that block for both jobs and ad hoc commands. It defaults to false, so nothing changes for existing inventories. Syncs are not affected: an ad hoc command never blocked a sync, and a running sync still blocks jobs unless allow_jobs_while_syncing is also set. The scheduler reads both inventory flags in a single query, once per busy inventory and cycle, so the number of queries does not grow with the number of pending tasks. Signed-off-by: fernandorocagonzalez <fernandorocagonzalez@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
SUMMARY
While an ad hoc command is running against an inventory, every job and every other ad hoc command launched against that inventory stays pending until it finishes:
That's the right default when the ad hoc command changes things under whatever comes next, like a reboot. But many ad hoc commands are long and harmless for everything else, a
dnf updateacross a few hundred hosts for example, and while they run they hold back scheduled jobs that have nothing to do with them.This adds an opt-in flag on the inventory,
allow_jobs_while_ad_hoc, that lifts that block. It's set by whoever administers the inventory, the same asallow_jobs_while_syncingfrom #1054, so jobs and their variables have no say in it.How it works:
Inventory.allow_jobs_while_ad_hoc, defaultfalse, so nothing changes for existing inventories. It's on the inventory and constructed inventory serializers and it's copied along with the inventory.DependencyGraph.inventory_blocked_by, a running ad hoc command no longer blocks when the inventory has the flag. It lets through both jobs and other ad hoc commands, since the reason to block them is the same.inventory_update_blocked_byonly looks at other syncs of the same source), and a running sync still blocks jobs and ad hoc commands unlessallow_jobs_while_syncingis also set. Each flag only lifts its own block. There are tests for both so this doesn't change by accident.allow_jobs_while_syncingcache from feat: let jobs run against an inventory while it is being synced #1054. The query count doesn't grow with the number of pending tasks, and there's a test that checks it.0235, depending on0233. Optionally keep only inventory variables provided by inventory sources #1202 uses0234, so whichever of the two goes in second needs renumbering, I'll take care of it.UI: an Allow Jobs While Ad Hoc Commands Run checkbox under Options in the inventory form and the constructed inventory form, listed under Enabled Options in both detail views, with help text. Spanish translation added, the other locales re-extracted.
Docs:
docs/inventory/allow_jobs_while_ad_hoc.md, andallow_jobs_while_syncing.mdnow points to it where it says ad hoc commands still block.ISSUE TYPE
COMPONENT NAME
ASCENDER VERSION
ADDITIONAL INFORMATION
Tests added:
TestAllowJobsWhileAdHocintest_scheduler.py: an ad hoc command blocks jobs and other ad hoc commands by default (with the right job explanation); with the flag both start; the flag of another inventory doesn't count; a sync starts while an ad hoc command runs, with the flag on or off; a running sync still blocks withoutallow_jobs_while_syncing, andallow_jobs_while_syncingalone doesn't let tasks past an ad hoc command; the flags are read once per cycle.Checks from
.github/copilot-instructions.md, run inside the dev container:End to end on the docker-compose dev environment, with an inventory holding one local host, a
sleep 150ad hoc command and a ping job template:waiting for adhoccommand-5 to finish. They start as soon as it ends.sleep 45ad hoc command running on it: a sync of its source starts and finishes during the ad hoc command, with the flag off and with it on, the same as before this change.