Skip to content

Track expected pipelines (pytorch/jax enable-flags) for finalize rollup #57

Description

@HereThereBeDragons

Sibling to #56. Same root cause, one level up: #56 tracks expected archs
within a pipeline; this tracks the expected set of pipelines.

Right now rebuild_summary decides pipeline status from which leaves have
reported. pytorch and jax are fire-and-forget workflow-dispatch triggers,
gated on inputs.build_pytorch / inputs.build_jax. As such the orchestrator
will not wait for the results of those pipelines. It can finalize
(completed_at) while a pytorch/jax leaf has never reported.

That makes a no-leaf pytorch/jax pipeline at finalize ambiguous between two
states the detail tree can't distinguish:

  • disabled (build_pytorch: false) → should render skipped
  • dispatched but not reported yet → should stay in_progress (correctly
    holds the release)

Today both render in_progress, so a finalized release with pytorch/jax
disabled shows those children in_progress forever while overall_status is
success.

Side note: native packages and rocm build do not have this problem as
they do not have an if-guard and are always triggered via workflow call by
the orchestrator, so the orchestrator waits for their results.

The signal exists

Either via the job list ("conclusion": "skipped") of the platform
orchestrator (multi_arch_release_linux.yml) or via the workflow inputs of the
same workflow.

overall_status must consider this

Today completed_at is stamped when the top-level orchestrator run
finishes - but that run does not wait on the dispatched pytorch/jax runs, so
completed_at does not mean those pipelines are done. Any consumer (and the
rollup itself) treating orchestrator completed_at as "release done" will
finalize prematurely for dispatched pipelines.

Once we track the expected pipeline set, this resolves naturally:

  • An enabled pytorch/jax that hasn't reported stays in_progress, feeds the
    platform worst-of, and holds overall_status at in_progress until it
    reports a terminal leaf - regardless of the orchestrator's completed_at.
  • A disabled pytorch/jax renders skipped and does not hold the release.

So overall_status should only roll up to a finalized state (success /
failure / cancelled) once every expected pipeline has reached a terminal
state — not merely when the top-level orchestrator's completed_at is set. The
completed_at cap in rebuild_summary (the branch that currently drops the
unstarted-pipeline injection once finalized) needs to key off the expected set
instead of orchestrator completion alone.

Note

In practice the enabled-pipeline case is usually masked: the dispatched
pytorch/jax run reports a started leaf (holding overall_status at
in_progress) before the top-level orchestrator finalizes, because other
in-flight workflow_call work keeps the orchestrator alive long enough. But
that ordering is a timing coincidence, not a needs: guarantee - a fast
release, slow dispatch pickup, or a disabled pipeline (no leaf at all,
timing-independent) still exposes the inconsistency. Without the expected-set
tracking, overall_status can also flap: in_progress --> success
(orchestrator finalizes, pytorch silent) --> in_progress (pytorch dispatch
lands) --> success. This issue hardens the logic so correctness does not depend
on that slack.

Relates to #56, #41.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions