Fork Sync: Update from parent repository#172
Merged
Merged
Conversation
Record the project workflow as a golden rule in the agent orientation guide: topic branches must be pushed to a personal fork remote and the pull request opened from there, never pushed directly to the shared upstream `origin` (openpmix/prrte). CLAUDE.md previously duplicated the entire orientation guide and had to be kept in sync with AGENTS.md by hand. Replace it with a symlink to AGENTS.md so the two can never drift. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
Testing PRRTE's elastic-DVM grow and shrink paths requires a multi-node DVM, and the only place those paths can be exercised without a real PMIx scheduler has been an ad-hoc set of containers rebuilt by hand each session. Capture that setup in the repo so a future contributor can clone upstream and bring up a working test swarm quickly. contrib/dockerswarm/ provides a build script, a Dockerfile, a four-node docker-compose definition, the PMIx tool client that drives the size changes, and a README documenting the workflow and its pitfalls. build.sh exports this repo's committed tree via git archive (so the image contains exactly the code under test, with no host build artifacts and no architecture mismatch) and the Dockerfile clones PMIx and compiles both into the image. The README calls out the one non-obvious requirement that otherwise silently wastes time: the DVM must be started with --prtemca prte_elastic_mode 1, or grow and shrink never complete. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org> (cherry picked from commit b4e427c)
Author
|
Hello! The Git Commit Checker CI bot found a few problems with this PR: 049db08: contrib: add a Docker harness for elastic-DVM grow...
Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks! |
Add docs/plans/ as a new top-level section for in-progress feature implementation plans. Move and update the DVM launch-fence plan (previously an untracked file under docs/how-things-work/) to docs/plans/elastic_dvm/dvm_launch_fence.rst. Also commit the companion state_machine.rst (was untracked) and wire both new sections into the appropriate index files. Signed-off-by: Ralph Castain <rhc@pmix.org>
Introduce a global launch fence that serialises app-job dispatch
with in-progress daemon grow and shrink campaigns, closing two race
windows:
Race 1 – a new app job reaches VM_READY → MAP while daemons are
still launching (grow) or terminating (shrink). It could be mapped
to a node whose daemon does not yet exist or is dying.
Race 2 – an app job completes MAP before a shrink is initiated and
then enters prte_plm_base_launch_apps() while a targeted daemon
is dying between MAP and the send.
The mechanism is a single integer counter (prte_dvm_launch_fence)
that is incremented at the start of every daemon launch campaign
(+1 per grow, +m per shrink targeting m daemons) and decremented
back to zero as each campaign completes. When the fence is nonzero,
app jobs are parked in one of two held-job arrays and released when
the fence reaches zero.
prte_shrink_campaigns is a pmix_list_t of prte_shrink_campaign_t
entries (one per PMIX_ALLOC_RELEASE request), each tracking the
array of target daemon ranks and a pending-ACK count. Multiple
concurrent shrink campaigns are fully supported.
-- New types and globals --
prte_globals.h / prte_globals.c
prte_dvm_launch_fence – running count of open campaign slots
prte_held_jobs – jobs parked at the VM_READY → MAP gate
prte_prelaunch_held_jobs – jobs parked at the LAUNCH_APPS gate
prte_shrink_campaign_t – pmix_list_item_t subtype carrying the
target rank array, ntargets, and the
pending-ACK count; destructor frees the
array
prte_shrink_campaigns – pmix_list_t of active shrink campaigns
prte_init.c – construct both pointer arrays and the campaign list
prte_finalize.c – destruct all three
-- New PLM type constants --
plm_types.h
PRTE_JOB_STATE_WAITING_FOR_DAEMONS (17)
Marker state set on parked jobs so that debug output makes the
reason for the park visible. No callback is registered for it.
PRTE_PLM_SHRINK_ACK_CMD (7)
PLM command sent daemon → HNP over PRTE_RML_TAG_PLM to confirm
that the daemon has exited due to a shrink request.
error_strings.c – add "WAITING FOR DAEMON CAMPAIGN" for the new state
-- New PLM base functions --
plm_private.h declares; plm_base_launch_support.c defines:
prte_plm_base_fence_release(bool success)
Called when prte_dvm_launch_fence reaches zero. Drains
prte_held_jobs (re-activating VM_READY or NEVER_LAUNCHED),
drains prte_prelaunch_held_jobs (re-activating LAUNCH_APPS,
remapping via MAP if any of the job's procs were assigned to a
shrink target, or NEVER_LAUNCHED on failure), and does a safety
sweep of any residual campaign entries.
prte_plm_base_job_needs_remap(jdata)
Returns true if any proc in jdata was mapped to a node whose
daemon rank appears in any active shrink campaign.
prte_plm_base_reset_proc_map(jdata)
Removes all proc objects from their nodes, decrementing both
node->num_procs and node->slots_inuse (mirroring the +1/+1
done by prte_rmaps_base_claim_slot; tool procs, which do not
count against slots_inuse, are handled with the PRTE_APP_FLAG_TOOL
check). Clears jdata->num_procs and jdata->num_launched so the
mapper can re-create proc objects on surviving nodes.
-- Grow fence --
plm_base_launch_support.c – prte_plm_base_setup_virtual_machine
Increment prte_dvm_launch_fence inside the existing
"if (0 < map->num_new_daemons)" block, after setting
PRTE_JOB_LAUNCHED_DAEMONS. All four PLM backends (ssh, slurm,
pals, lsf) call this common path, so one insertion covers every
extension scenario. The attribute and the fence are always set
on the daemon job.
state_dvm.c – vm_ready
The fence decrement is placed just before the closing brace of
the outer PRTE_JOB_LAUNCHED_DAEMONS block, after the inner
"!DO_NOT_LAUNCH && num_daemons > 1" xcast block. This covers
both the xcast path and the DO_NOT_LAUNCH / single-daemon path.
Each inner error-exit path (nidmap_create, PMIx_Get, two
PMIx_Data_pack calls, and the xcast itself) gets its own
fence decrement + fence_release(false) before calling
PRTE_ACTIVATE_JOB_STATE(NULL, FORCED_EXIT).
Before preposition_files (the app-job branch), park the job in
prte_held_jobs when prte_dvm_launch_fence > 0 and set its state
to PRTE_JOB_STATE_WAITING_FOR_DAEMONS.
state_dvm.c – check_complete
Inside the "if (NULL == jdata || PMIX_CHECK_NSPACE(...))" block
(the daemon-job-only path), check whether PRTE_JOB_LAUNCHED_DAEMONS
is set and prte_dvm_launch_fence > 0 before any of the early
returns. If so, decrement and call fence_release(false). This
is the safety net for the case where the daemon job reaches
TERMINATED rather than going through the vm_ready error paths.
errmgr_dvm.c – proc_errors
After marking a daemon as gone and decrementing num_daemons,
check whether a grow campaign is in progress
(prte_dvm_launch_fence > 0 && PRTE_JOB_LAUNCHED_DAEMONS set
on the daemon job). If so, decrement the fence, call
fence_release(false) if the fence hits zero, and remove the
PRTE_JOB_LAUNCHED_DAEMONS attribute to prevent double-decrement
if a second daemon in the same campaign crashes.
-- Shrink fence --
ras_base_allocate.c – prte_ras_base_complete_request (PMIX_ALLOC_RELEASE)
Before free(ranks) (which precedes the xcast): allocate a
prte_shrink_campaign_t, memcpy the rank array into it, set
ntargets = pending = m, append to prte_shrink_campaigns, and
increment prte_dvm_launch_fence by m. If the subsequent xcast
fails, remove the just-appended campaign from the list and
restore the fence to prevent held jobs from parking forever.
prted_comm.c – PRTE_DAEMON_SHRINK_CMD handler
After PRTE_PMIX_WAIT_THREAD(&lk) completes the JOB_END
notification but before activating DAEMONS_TERMINATED:
allocate a buffer with PMIX_DATA_BUFFER_CREATE, pack
PRTE_PLM_SHRINK_ACK_CMD and PRTE_PROC_MY_NAME->rank, and send to
PRTE_PROC_MY_HNP->rank on PRTE_RML_TAG_PLM using
PRTE_RML_RELIABLE_SEND. The rank is included so the HNP can
match the ACK to the correct campaign entry without relying on
sender metadata.
plm_base_receive.c – PRTE_PLM_SHRINK_ACK_CMD case
Unpack the sender rank, iterate prte_shrink_campaigns to find
the entry whose targets array contains that rank, decrement
campaign->pending and prte_dvm_launch_fence, remove and release
the campaign entry when pending hits zero, and call
fence_release(true) when the fence hits zero. Logs unrecognised
ranks at verbose level 1.
plm_base_launch_support.c – prte_plm_base_launch_apps
After the job-state guard and state update, check
!pmix_list_is_empty(&prte_shrink_campaigns) (not
prte_dvm_launch_fence > 0, so that a concurrent grow does not
hold jobs already mapped to surviving nodes). If nonempty, set
PRTE_JOB_STATE_WAITING_FOR_DAEMONS, PMIX_RETAIN the job, add it
to prte_prelaunch_held_jobs, release the caddy, and return.
errmgr_dvm.c – proc_errors (shrink crash fallback)
After the grow-fence check, iterate prte_shrink_campaigns with
PMIX_LIST_FOREACH_SAFE looking for the dead daemon's rank. When
found, decrement campaign->pending and prte_dvm_launch_fence,
remove and release the campaign if pending reaches zero, and call
fence_release(true) if the fence reaches zero. This prevents
held jobs from parking indefinitely when a shrink target crashes
before it can send its ACK.
Signed-off-by: Ralph Castain <rhc@pmix.org>
Fix unknown Pygments lexer for m4 code block by using 'autoconf' instead. Signed-off-by: Ralph Castain <rhc@pmix.org>
A shrink campaign previously completed when each targeted daemon sent a PRTE_PLM_SHRINK_ACK_CMD to the HNP just before exiting, with the errmgr comm-failure path kept only as a fallback for daemons that crashed before acking. That design was unsound. An ack announces a daemon's intent to leave while it is still a live participant in the DVM, but the state the launch fence protects against -- a job being mapped onto, or having launch data sent to, a departing daemon -- is only safe once the daemon's routes, num_daemons count, and node state have been torn down. That teardown happens on the comm-failure path, not when the ack is sent, so releasing held jobs on ack receipt could unpark them into a DVM that still believed the departing daemon was present. Worse, the two decrement paths double-counted. Because nothing marked a target as already counted and a campaign was only removed once its pending count hit zero, a target could be decremented once when its ack arrived and again when its subsequent death was detected. For a two-target campaign the first daemon's ack-then-death could drive pending to zero on its own, completing the campaign and releasing the fence while the second target was still present -- reopening the very race the fence exists to close. The reason a daemon departs carries no useful information: a clean shrink exit and a crash have identical consequences, since the node is being removed either way and the processes beneath the daemon die with it. So the ack is removed entirely -- the daemon-side send, the PRTE_PLM_SHRINK_ACK_CMD constant, and the HNP-side handler. Campaign completion is now driven solely by actual daemon departure on the comm-failure path, which is both the authoritative event and the point at which the relevant cleanup has occurred. Each matched target slot is stamped PMIX_RANK_INVALID once counted so a daemon that emits more than one failure event cannot decrement twice. The implementation plan and state-machine documentation are updated to match, and a design-decision section records why the ack approach was abandoned. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
A RAS component that handles a grow or shrink request and fires PRTE_JOB_STATE_LAUNCH_DAEMONS itself bypasses the bookkeeping that the DVM launch fence depends on: setting PRTE_JOB_EXTEND_DVM and resetting the nidmap flag on grow, and recording the shrink campaign and raising prte_dvm_launch_fence before any daemon is asked to leave on shrink. Skipping that common handling leaves the fence out of step with the campaign it is meant to gate, reopening the daemon-launch race. Document that modify results must be routed through prte_ras_base_complete_request(), and note that the Slurm scheduler-push path currently fires the state directly, so readers understand the hazard and new modules follow the correct pattern. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
The DVM grow (daemon-launch) path encoded an in-progress launch as a single PRTE_JOB_LAUNCHED_DAEMONS boolean on the daemon job with a one-per-campaign launch-fence bump. Because that token carried no identity, two faults followed. An unrelated daemon dying while a grow was in flight consumed the campaign's token: it released the jobs held at the VM_READY to MAP boundary early, reopening the race the fence exists to close, and cleared the token so vm_ready then skipped the WIREUP xcast, leaving the genuinely new daemons unwired. Two overlapping grows shared the single token, so a failure could leave the fence stuck above zero and the held jobs parked indefinitely. Track each grow campaign explicitly, recording the specific daemon ranks being launched in a prte_grow_campaign_t, mirroring the rank-tracked prte_shrink_campaign_t already used by the shrink path. The campaign's fence contribution is held in full until the campaign is drained as a unit: on success in vm_ready, after the WIREUP xcast, so jobs held at VM_READY are admitted only once the new daemons are wired up; or on failure when one of the launched ranks dies. A daemon loss only affects the fence when its rank belongs to an in-progress campaign, so unrelated deaths no longer consume it, and concurrent campaigns are independent objects that cannot wedge the fence. A separate prte_grow_campaigns list is used rather than unifying with prte_shrink_campaigns because the LAUNCH_APPS hold and remap-on-release logic must remain shrink-only; a grow must not stall jobs already mapped onto existing nodes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
The launch-fence plan still described the grow path's fence accounting as a single PRTE_JOB_LAUNCHED_DAEMONS boolean plus a bare prte_dvm_launch_fence increment and decrement, with per-error-exit decrements in vm_ready, the errmgr, and check_complete. That design was superseded by the per-campaign, rank-tracked accounting now documented in the grow-campaign plan and implemented in the tree, so the old text contradicted the code: error-exits deliberately leave the fence to DVM teardown, failures route through prte_plm_base_grow_target_failed(), and check_job_complete drains via prte_plm_base_grow_drain(). Reduce the grow-specific steps, the grow file-change table, and the grow design invariants to pointers into the grow-campaign plan, keeping only the shared fence mechanism and the shrink-path content that remain accurate. Also fix a dangling cross-reference that pointed at a nonexistent Step 12. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
A rebase conflict resolution left the implementation-plans toctree listing per_app_mapping/index.rst twice, which Sphinx flags as a duplicate document reference. Remove the stray second entry. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
The elastic_dvm plan directory carried only implementation plans for the DVM launch fence and grow-campaign tracking, with no companion document stating the externally observable contract those mechanisms uphold. The node_reservation directory pairs each plan with such a spec, and the same separation is wanted here: a reader (or a future implementor) needs to know what the runtime guarantees a caller, independent of how the fence and campaign accounting deliver it. Add elastic_dvm_spec.rst describing the job-admission and placement guarantees in effect while the DVM grows or shrinks: non-destructive deferral of a job submitted during a size change, the two placement hazards that are closed, the remap-onto-survivors behavior, completion driven by actual daemon departure, the failure and concurrency semantics, and the single observable artifact (the WAITING_FOR_DAEMONS job state). List it first in the directory index and have both plans reference it as authoritative for observable behavior. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
The elastic-DVM admission spec previously said only that a job whose grow dependency fails is aborted; it was silent on the fate of the daemons the failed grow had already started. Leaving them running would strand the DVM in a half-extended state — a partial, un-wired set of new daemons that no job can safely use and that no later event reclaims. Specify that a failed grow is rolled back as a whole: the campaign's already-started daemons are terminated and the nodes it was adding leave the DVM, restoring the exact membership the DVM had before the campaign began. The rollback is scoped to the failed campaign, so a concurrent grow and all pre-existing daemons are untouched. Record the guarantee in the spec's grow-failure behavior, failure-semantics table, scope, and conformance summary, and reconcile grow_campaign.rst with it: the targets array now also enumerates the daemons to tear down, the failure drain step rolls the campaign back via the shrink-path machinery before draining held jobs, and a new section details the teardown including the FAILED_TO_START edge case. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
PR #2472 review observed that draining a shrink campaign one daemon at a time drives a separate routing-tree repair per departure, so removing several daemons along one branch of the radix tree can trigger repeated promotions that may prove costly for a large shrink. Record the suggested collective alternative as an explicit follow-up in the launch-fence plan: confirm the broadcast once, then report all of the campaign's targets as failed in a single batch repair, with the doomed daemons exiting on lifeline disconnect rather than self-exiting. Explain why this preserves the plan's "act on teardown, not on intent" invariant and is therefore distinct from the per-daemon acknowledgement the plan already rejected, and why it is deferred out of the launch-fence work: it is an optimization rather than a correctness fix, needs a new xcast-completion callback, relocates the completion trigger, and exercises the less-tested whole-branch tree-repair path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
A dynamic allocation that grows or shrinks the DVM is answered as soon as the request has been processed, not when the resulting daemon launch or teardown actually finishes. The spec previously described only the job-admission side of a size change and was silent on how the requester learns that the asynchronous operation completed — leaving "the allocation is complete" and "the runtime is ready" conflated. Separate the two points explicitly. Specify a two-phase model: a synchronous response on request acceptance, followed by exactly one asynchronous, requester-directed terminal event when the DVM operation finishes — PMIX_DVM_IS_READY on success and PMIX_ERR_DVM_MOD, carrying the underlying cause, on failure. Define each event's allocation-identifying payload, the delivery guarantees (no event for a phase-one rejection, for a no-op request, or for a requester-less scheduler push), and propose the two new PMIx event codes the contract requires, gated by a PMIx capability check. Broaden the document accordingly: retitle it from "Job Admission" to cover both contracts, add the supporting definitions, and record the clean degradation when the underlying PMIx lacks the codes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
The single dvm_launch_fence.rst plan mixed the shared launch-fence infrastructure with the shrink path's specifics while the grow path lived in its own file, an asymmetry that made the shrink design hard to find and left no parent tying the three together. The spec also now requires a phase-two completion event that neither plan described emitting. Restructure the plan tree for symmetry and reconcile it with the spec. Rename dvm_launch_fence.rst to elastic_dvm_plan.rst as the parent, holding only the shared mechanism — the fence counter, held-job arrays, the VM_READY hold point, the fence-release helper, and a new shared completion-event helper — and linking to the two campaign plans. Extract the shrink-specific design into shrink_campaign.rst alongside the existing grow_campaign.rst, renumbering its steps and pointing its shared references at the parent. Thread the spec's two-phase completion contract through both campaign plans: each campaign object records the requester, allocation id, and request id, and emits PMIX_DVM_IS_READY when its operation completes or PMIX_ERR_DVM_MOD (with the underlying cause) when it fails, via a shared prte_plm_base_dvm_mod_notify() helper gated by a PMIx capability check. Update the index toctree and retarget the spec and state-machine cross-references at the new labels. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
…p flag The plan and spec said the PMIX_DVM_IS_READY / PMIX_ERR_DVM_MOD event codes would be gated by the PRTE_CHECK_PMIX_CAP capability-flag idiom. That is heavier than needed: PMIx status codes are plain preprocessor "#define"d pmix_status_t values, so whether the installed PMIx supplies a code is decided entirely by the symbol's presence in the headers, which the preprocessor can test directly. A PMIX_CAP_* flag is for behavioral features a single compile-visible symbol cannot capture; PRRTE only emits these events and a receiver opts in by registering for the code, so no such flag applies. Reword both documents to guard the helper and its call sites with a simple preprocessor existence check. To keep the project's "#if FOO" discipline, recommend a one-line m4 probe that sets PRTE_HAVE_DVM_MOD_EVENTS to 0/1 from the symbols' presence, while noting a direct "#if defined(...)" guard is equally correct, and mark the config/prte_setup_pmix.m4 touch as optional. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
The plan left the completion-event guard as a choice between a PRTE_HAVE_DVM_MOD_EVENTS macro and a direct "#if defined(...)". Settle on the macro form so the project keeps a single "#if FOO" guard whose typo is a compile error, and make the plan prescriptive: give the concrete AC_PREPROC_IFELSE probe that defines PRTE_HAVE_DVM_MOD_EVENTS to 0 or 1 from the symbols' presence, drop the "optional" marking on the config/prte_setup_pmix.m4 touch, and note the autogen.pl re-run. Localise the guard at the same time. Give the shared notify helper a "bool success" parameter instead of an event-code argument, so the two new status codes are named only inside the helper body; call sites pass a bool and a cause and need no guard, leaving just the helper body under "#if PRTE_HAVE_DVM_MOD_EVENTS". Update the grow and shrink call sites accordingly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
The elastic-DVM spec promises (conformance guarantee #4) that a daemon failure affects only the jobs waiting on the campaign that failure belongs to. The implementation plan released held jobs through a single prte_plm_base_fence_release(bool success) keyed on the global fence reaching zero, which could not honor that promise when a grow failed while a shrink was still in progress: a single global success flag cannot express "abort the grow's waiters but leave the shrink's waiters parked," and deferring the abort until the fence reached zero let a later shrink-success release admit a job whose grow dependency had already failed. Split the release into a success-only prte_plm_base_fence_release() and a grow-failure prte_plm_base_abort_premap_held() that fails only the pre-map held jobs, immediately and independent of the fence, leaving the shrink-only pre-launch held jobs parked. Pre-map jobs wait on every in-progress campaign including the grow; pre-launch jobs wait only on a shrink, so this is exactly what guarantee #4 requires. Also guard shrink-campaign creation on a nonzero target count so an empty release cannot leave prte_shrink_campaigns permanently non-empty and wedge every later job at the LAUNCH_APPS hold, mirroring the grow path's num_new_daemons guard; consolidate the grow helpers into plm_base_launch_support.h with the other launch-fence helpers; and have the defensive campaign sweep cover both the grow and shrink lists. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
The launch-fence plans named src/mca/plm/base/plm_base_launch_support.h as the home for the shared and per-campaign helper declarations, but no such header exists. The shared helpers are called from the errmgr, ras, and state components, which live outside plm-base and therefore include the public plm-base header src/mca/plm/base/base.h (where the existing prte_plm_base_* entry points such as prte_plm_base_launch_apps are declared); plm_private.h is private to plm-base and not visible to them. Point every declaration reference at base.h so the plan matches the tree and the callers can actually see the prototypes. The definitions remain in plm_base_launch_support.c, which does exist. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
The previous commit moved the helper declarations to base.h on the premise that the errmgr, ras, and state callers could not otherwise see them. That premise was wrong: the existing launch-fence implementation already declares these helpers in src/mca/plm/base/plm_private.h, and errmgr_dvm.c and state_dvm.c already include that header to reach the existing prte_plm_base_* entry points. Point the plans at the header the code actually uses, so the documentation matches the tree. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
…ease Bring the launch-fence implementation up to the revised elastic-DVM plans. Two things were missing relative to the spec, and one shared helper conflated outcomes it must keep separate. First, the two-phase completion contract had no realization: an accepted grow or shrink returned its synchronous allocation response but never told the requester, asynchronously, that the DVM operation had actually finished. Add prte_plm_base_dvm_mod_notify(), which directs a PMIX_DVM_IS_READY event to the requester on success or PMIX_ERR_DVM_MOD (carrying the underlying cause) on failure, mirroring the custom-range delivery used for PMIX_ALLOC_TIMEOUT_WARNING. The two status codes are plain #defines, so a configure probe defines PRTE_HAVE_DVM_MOD_EVENTS from their presence in the installed PMIx headers; when absent the helper compiles to a no-op and the runtime simply omits the notification. Both campaign objects now record the requester / PMIX_ALLOC_ID / PMIX_ALLOC_REQ_ID; the shrink path captures them from the originating PMIX_ALLOC_RELEASE and emits on completion (errmgr) and on xcast-failure (ras). The grow path carries the fields but leaves have_requester false for now, because the originating allocation request is not yet threaded through to setup_virtual_machine; wiring that is a follow-up and does not affect fence accounting. Second, prte_plm_base_fence_release(bool success) could not honor the spec when a grow failed while a shrink was still in progress: a single global success flag cannot abort the grow's waiters while leaving the shrink's waiters parked, and gating the failure on the fence reaching zero let a later shrink-success release admit a job whose grow dependency had already failed. Split it into a success-only fence_release() and a grow-failure abort_premap_held() that fails only the pre-map held jobs, immediately and independent of the fence; the pre-launch (shrink-only) held jobs are never touched by a grow failure, per the spec's campaign-scoped failure guarantee. Also guard shrink-campaign creation on a nonzero target count so an empty release cannot leave the campaign list non-empty forever and wedge later jobs at the LAUNCH_APPS hold, and have the defensive campaign sweep cover both the grow and shrink lists. Because config/prte_setup_pmix.m4 changed, autogen.pl must be re-run. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
The grow campaign carried the requester fields for the phase-two completion event but always left have_requester false, so a grow emitted no PMIX_DVM_IS_READY / PMIX_ERR_DVM_MOD even when it was driven by a dynamic allocation request. The missing link was that the originating request is not passed to setup_virtual_machine(), where the campaign is created. It does not need to be passed explicitly: the RAS reservation machinery already records the driving request on the session and back-points every reserved node at it (add_nodes_to_session sets node->session, and the session carries requestor, alloc_refid, and user_refid). Read those from the first new daemon's node when building the grow campaign. The initial DVM bring-up and a scheduler-driven push have no such requestor — the default session, or an invalid requestor rank — so have_requester stays false and no event is emitted for them, exactly as the spec requires. With this, both the grow and shrink paths deliver the spec's phase-two completion event to the process that requested the size change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
Update the grow and shrink plans to describe what was actually built. The grow campaign records its requester indirectly: the RAS reservation machinery back-points each reserved node at the session that carries the driving request, so setup_virtual_machine() reads the requestor and the allocation ids from the first new daemon's node->session rather than having the request threaded into the launch path. Document that, and the parallel shrink capture from req->tproc / req->info, replacing the earlier pseudocode that referenced locals the code does not have and line numbers that have since drifted. Note that the failure cause in PMIX_ERR_DVM_MOD is carried under PMIX_JOB_TERM_STATUS, and that the completion event is delivered as a directed custom-range notification. Most importantly, mark the grow failure rollback as not yet implemented: grow_target_failed() currently only drains the fence and aborts the pre-map held jobs, so a failed grow can still leave the DVM half-extended and does not yet satisfy the spec's conformance guarantee #5. The "Rollback on failure" section is retained as the specification for that remaining work and is now the primary item in the grow Follow-up. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
A grow that lost a daemon previously drained the fence and aborted the held jobs but left the partially-launched daemons running, so a failed grow could leave the DVM half-extended — violating the spec's conformance guarantee #5. Worse, the errmgr's general daemon-loss path would abort the entire DVM when a routeless new daemon failed to start, killing a running DVM over a grow that should simply have been rolled back. Make grow_target_failed() roll the matched campaign back out of the DVM. The new grow_rollback() repairs routing for the triggering rank, then for each other target either terminates it with the same PRTE_DAEMON_SHRINK_CMD xcast the shrink path uses (when the daemon actually started) or reverts its launch-time daemon-count bump (when it never started), and clears each node's daemon backpointer so the mapper can no longer place a job there — the new nodes carry no application procs, since the jobs that would have used them were held at the fence and never launched. The teardown is campaign-scoped, so a concurrent grow keeps its daemons and completes normally. grow_target_failed() now returns whether the loss was a grow target, and the errmgr gotos its cleanup on true so the general daemon-loss handling — which would abort the whole DVM — is skipped; the rollback has already absorbed the loss. The failure is now per-campaign rather than draining every grow at once, so an unrelated concurrent grow is no longer dragged down with the failed one. Two edges remain in the rarely-exercised launch-failure path, documented in the plan: a slow-to-start target is treated as never-started, and nodes are detached via node->daemon rather than removed from prte_node_pool (so num_nodes is not decremented), matching how the shrink path leaves the pool. None of this is yet validated against a real multi-node allocation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
The launch-fence, campaign-tracking, and grow/shrink completion machinery was active on every DVM. That changed fixed-size DVM behavior: initial bring-up launches daemons, which created a grow campaign and raised the fence, and a daemon loss during a grow was then absorbed as a campaign event — most consequentially, grow_target_failed() would report the loss as handled and the errmgr would skip its normal abort, so a daemon dying on an ordinary DVM no longer brought the DVM down as it used to. Gate the whole feature on the pre-existing prte_elastic_mode MCA parameter (off by default), which already governs the related decision to pass SLURM --no-kill so daemon losses are tolerated. The gate is applied where the fence is raised — grow-campaign creation in setup_virtual_machine() and shrink-campaign creation in the PMIX_ALLOC_RELEASE handler — so outside elastic mode the fence stays zero and the campaign lists stay empty, leaving every downstream check inert. The consumer sites (the VM_READY and LAUNCH_APPS holds, the vm_ready drain, and grow_target_failed()) also carry an explicit guard so the non-elastic path is provably identical to the pre-feature behavior: grow_target_failed() now returns false immediately when elastic mode is off, so a daemon loss on a fixed-size DVM follows the ordinary errmgr abort path exactly as before. The PMIX_ALLOC_RELEASE xcast still fires when not in elastic mode, so an explicit release retains its legacy fire-and-forget behavior; only the campaign/fence/parking is suppressed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
The grow path's phase-two completion event (PMIX_DVM_IS_READY) was never delivered to the requester. prte_ras_base_complete_request() registered the newly reserved nodes with their destination session by calling add_nodes_to_session(&ndlist, dest) -- but it did so only after prte_ras_base_node_insert(&ndlist, NULL), which drains ndlist (it removes and releases every item as it folds them into the global node pool). The registration loop therefore iterated an empty list and set no session backpointer at all. That backpointer is the sole carrier of the requestor identity from the RAS reservation across to the launch-side grow campaign: setup_virtual_machine reads the new daemon's node->session to populate the campaign's requester, and grow_drain() emits the completion event only when a requester is present. With the backpointer perpetually NULL, have_requester stayed false and the event was silently dropped, leaving any client that issued the allocation request waiting forever for a completion it would never receive. Snapshot the node names before the insert and resolve the pool objects by name in add_nodes_to_session(), which already used prte_node_match() rather than the list objects themselves. This was found and verified with a multi-node container DVM: the requester now receives PMIX_SUCCESS followed by PMIX_DVM_IS_READY. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
prte_shrink_campaign_t and prte_grow_campaign_t were declared with a NULL constructor, so PMIX_NEW returned an object whose alloc_id and req_id pointers held uninitialized memory. The shrink creator in prte_ras_base_complete_request sets those two fields only when the release request actually carries PMIX_ALLOC_ID / PMIX_ALLOC_REQ_ID; a release that names nodes but omits one of them therefore left the corresponding pointer pointing at garbage. When the campaign was later released, campaign_destruct() called free() on that garbage pointer and corrupted the heap, aborting the HNP with a malloc consolidation error mid-shrink. The grow path happened to escape this only because setup_virtual_machine assigns every field by hand. Give both campaign classes a constructor that NULLs targets, alloc_id, and req_id (and zeroes the counts and requester) so an unset identifier is a safe free(NULL) rather than a wild pointer. Found with a multi-node container DVM: releasing a node now leaves the HNP healthy and removes only the named daemon. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
A tool's PMIX_ALLOC_NEW/EXTEND/RELEASE request reaches the DVM master and is dispatched across the selected RAS modules by prte_ras_base_modify. The ras/pmix module forwards such requests to an external PMIx scheduler, but when none is attached prte_pmix_set_scheduler() fails and the module returned that hard error (e.g. PMIX_ERR_UNREACH) directly. Because the modify loop treats any non-SUCCESS, non-TAKE_NEXT status as a fatal error and breaks, that aborted the whole request before the lower-priority ras/hosts module was ever consulted -- so an elastic DVM running without a scheduler could not grow or shrink at all, and the only way to drive it was the ras/pmix "simulate" shortcut, which discards the request's node list and allocation ids and so cannot target a specific reservation for release. Make ras/pmix yield with PMIX_ERR_TAKE_NEXT_OPTION when no scheduler is reachable, allowing the loop to fall through to the next module, and teach ras/hosts -- the lowest-priority, last-resort local resource authority -- to claim the PMIX_ALLOC_NEW/EXTEND/RELEASE directives so the base complete_request logic runs with the original request info intact. Scheduler deployments are unaffected: ras/pmix still forwards whenever a scheduler is attached, and ras/hosts is reached only after it declines. This makes schedulerless grow and shrink testable with real node lists and reservation ids instead of the simulate shortcut. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
The ras/pmix simulate option faked a scheduler interaction by injecting a fixed node list (ras_pmix_simulate_nodelist) in place of the request, returning it as though a scheduler had supplied it. It was only ever a stand-in for exercising the elastic grow path without a real scheduler, and it was a poor one: because it overwrote the request, it discarded the caller's node list and allocation ids and so could not target a specific reservation for release. Now that ras/hosts satisfies node-list grow and shrink locally, with the original request info intact, the simulate shortcut is redundant. Remove the two MCA parameters (ras_pmix_simulate, ras_pmix_simulate_nodelist), their component fields, and the simulate branch in modify(); a schedulerless DVM is driven directly through ras/hosts instead. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
The shrink-campaign design doc carried a lengthy follow-up note describing a possible optimization: repairing the routing tree once per shrink campaign instead of once per departing daemon. This is an optimization rather than a correctness fix and is not part of the launch-fence work, so it does not belong in the design doc going forward. Move the note's content to #2492 so it is tracked as actionable work, and leave a short pointer to that issue in its place. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
A PMIX_ALLOC_RELEASE that shrinks the DVM can name nodes drawn from more than one underlying allocation, so no single RAS component owns the whole set of departing resources. To let each component hand back the share it manages, add a shrink_complete entry point to the ras module interface and a prte_ras_base_shrink_complete() helper that cycles across every active module, passing the completed shrink campaign so the module can identify which release finished. The errmgr drives this at the campaign-completion point: once the last targeted daemon has departed, it offers the campaign to every module before emitting the requester's completion notification. The runtime guarantees only the ordering — each component is given the opportunity to release before PMIX_DVM_IS_READY fires — not that any particular resource is returned, which remains the component's decision. No component implements the hook yet, so the cycle is currently a no-op. The elastic-DVM spec and plan are updated to document the multi-allocation case and this new step in the shrink workflow. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
Author
|
Hello! The Git Commit Checker CI bot found a few problems with this PR: 049db08: contrib: add a Docker harness for elastic-DVM grow...
Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.