Sibling to #56. Same root cause, one level up: #56 tracks expected archs
within a pipeline; this tracks the expected set of pipelines.
Right now rebuild_summary decides pipeline status from which leaves have
reported. pytorch and jax are fire-and-forget workflow-dispatch triggers,
gated on inputs.build_pytorch / inputs.build_jax. As such the orchestrator
will not wait for the results of those pipelines. It can finalize
(completed_at) while a pytorch/jax leaf has never reported.
That makes a no-leaf pytorch/jax pipeline at finalize ambiguous between two
states the detail tree can't distinguish:
- disabled (
build_pytorch: false) → should render skipped
- dispatched but not reported yet → should stay
in_progress (correctly
holds the release)
Today both render in_progress, so a finalized release with pytorch/jax
disabled shows those children in_progress forever while overall_status is
success.
Side note: native packages and rocm build do not have this problem as
they do not have an if-guard and are always triggered via workflow call by
the orchestrator, so the orchestrator waits for their results.
The signal exists
Either via the job list ("conclusion": "skipped") of the platform
orchestrator (multi_arch_release_linux.yml) or via the workflow inputs of the
same workflow.
overall_status must consider this
Today completed_at is stamped when the top-level orchestrator run
finishes - but that run does not wait on the dispatched pytorch/jax runs, so
completed_at does not mean those pipelines are done. Any consumer (and the
rollup itself) treating orchestrator completed_at as "release done" will
finalize prematurely for dispatched pipelines.
Once we track the expected pipeline set, this resolves naturally:
- An enabled pytorch/jax that hasn't reported stays
in_progress, feeds the
platform worst-of, and holds overall_status at in_progress until it
reports a terminal leaf - regardless of the orchestrator's completed_at.
- A disabled pytorch/jax renders
skipped and does not hold the release.
So overall_status should only roll up to a finalized state (success /
failure / cancelled) once every expected pipeline has reached a terminal
state — not merely when the top-level orchestrator's completed_at is set. The
completed_at cap in rebuild_summary (the branch that currently drops the
unstarted-pipeline injection once finalized) needs to key off the expected set
instead of orchestrator completion alone.
Note
In practice the enabled-pipeline case is usually masked: the dispatched
pytorch/jax run reports a started leaf (holding overall_status at
in_progress) before the top-level orchestrator finalizes, because other
in-flight workflow_call work keeps the orchestrator alive long enough. But
that ordering is a timing coincidence, not a needs: guarantee - a fast
release, slow dispatch pickup, or a disabled pipeline (no leaf at all,
timing-independent) still exposes the inconsistency. Without the expected-set
tracking, overall_status can also flap: in_progress --> success
(orchestrator finalizes, pytorch silent) --> in_progress (pytorch dispatch
lands) --> success. This issue hardens the logic so correctness does not depend
on that slack.
Relates to #56, #41.
Sibling to #56. Same root cause, one level up: #56 tracks expected archs
within a pipeline; this tracks the expected set of pipelines.
Right now
rebuild_summarydecides pipeline status from which leaves havereported. pytorch and jax are fire-and-forget
workflow-dispatchtriggers,gated on
inputs.build_pytorch/inputs.build_jax. As such the orchestratorwill not wait for the results of those pipelines. It can finalize
(
completed_at) while a pytorch/jax leaf has never reported.That makes a no-leaf pytorch/jax pipeline at finalize ambiguous between two
states the detail tree can't distinguish:
build_pytorch: false) → should renderskippedin_progress(correctlyholds the release)
Today both render
in_progress, so a finalized release with pytorch/jaxdisabled shows those children
in_progressforever whileoverall_statusissuccess.The signal exists
Either via the job list (
"conclusion": "skipped") of the platformorchestrator (
multi_arch_release_linux.yml) or via the workflow inputs of thesame workflow.
overall_statusmust consider thisToday
completed_atis stamped when the top-level orchestrator runfinishes - but that run does not wait on the dispatched pytorch/jax runs, so
completed_atdoes not mean those pipelines are done. Any consumer (and therollup itself) treating orchestrator
completed_atas "release done" willfinalize prematurely for dispatched pipelines.
Once we track the expected pipeline set, this resolves naturally:
in_progress, feeds theplatform worst-of, and holds
overall_statusatin_progressuntil itreports a terminal leaf - regardless of the orchestrator's
completed_at.skippedand does not hold the release.So
overall_statusshould only roll up to a finalized state (success/failure/cancelled) once every expected pipeline has reached a terminalstate — not merely when the top-level orchestrator's
completed_atis set. Thecompleted_atcap inrebuild_summary(the branch that currently drops theunstarted-pipeline injection once finalized) needs to key off the expected set
instead of orchestrator completion alone.
Note
In practice the enabled-pipeline case is usually masked: the dispatched
pytorch/jax run reports a
startedleaf (holdingoverall_statusatin_progress) before the top-level orchestrator finalizes, because otherin-flight
workflow_callwork keeps the orchestrator alive long enough. Butthat ordering is a timing coincidence, not a
needs:guarantee - a fastrelease, slow dispatch pickup, or a disabled pipeline (no leaf at all,
timing-independent) still exposes the inconsistency. Without the expected-set
tracking,
overall_statuscan also flap:in_progress-->success(orchestrator finalizes, pytorch silent) -->
in_progress(pytorch dispatchlands) -->
success. This issue hardens the logic so correctness does not dependon that slack.
Relates to #56, #41.