Make live Cloud integration CI selective, on-demand, and resilient

[sdairs]

Context

The live Cloud integration workflow is expensive, mutates a shared test organization, and currently runs on every matching PR update. This becomes particularly noisy and unreliable when refreshing a stack of PRs.

Example failure: https://github.com/ClickHouse/clickhousectl/actions/runs/31696362617/job/94434938976?pr=393

That job failed on a one-shot query-endpoint upsert with 503 SERVICE_UNAVAILABLE. It was not related to the PR changes. During the same stacked-PR refresh, many Cloud Integration jobs entered the shared organization together and produced:

• 429 TOO_MANY_REQUESTS while creating API keys, updating services, and cleaning up.
• Repeated 503 SERVICE_UNAVAILABLE responses from query-endpoint upserts.
• A transient Query API 404 saying the ClickHouse service was currently unavailable.
• Postgres 500 Internal error responses.

The linked job spent roughly four minutes compiling, failed about one minute into the live test, and then spent another 2.5 minutes cleaning up before reporting failure. Rerunning the job repeats all of that work.

Current behavior

.github/workflows/cloud-integration.yml runs five suites sequentially in one job:

1. ClickHouse service lifecycle.
2. Postgres lifecycle.
3. Organization lifecycle.
4. Postgres CDC ClickPipe.
5. Optional ClickPipe smoke tests.

The workflow currently triggers for every change under crates/clickhousectl/src/cloud/**, but it runs only cargo test -p clickhouse-cloud-api. CLI-only changes therefore launch live API tests that neither compile nor execute the changed CLI code.

The API source is now split into domain modules, but every API-domain change still launches every live suite. ClickStack, UDF, and backup changes also launch the workflow even though it has no live suite for those domains.

Existing resilience is limited:

• poll_until retries polling checks at a fixed interval, without jitter or error classification.
• retry_api_call exists but is used only for the known Postgres read-replica "no backups yet" race.
• Direct management operations normally get one attempt.
• Cleanup mutations normally get one attempt and can themselves fail with 429 or 5xx.
• The API test client has no explicit request timeout.
• Workflow concurrency is scoped to one PR ref, so it does not limit concurrent jobs from different PRs in a stack.

Goals

• Keep ordinary build, unit-test, clippy, and formatting CI automatic.
• Make live Cloud integration opt-in rather than running on every PR update.
• Allow one full run on the top PR of a stack to validate the complete inherited stack.
• Avoid running unrelated live suites when a targeted domain run is sufficient.
• Absorb temporary API throttling and service failures without rerunning a complete job.
• Keep cleanup reliable when the original test operation fails.

Proposed work

1. Harden live tests

• Add bounded exponential backoff with jitter for 429, retryable 5xx, and transport timeouts.
• Back off more aggressively after transient polling errors instead of keeping every job on the same fixed cadence.
• Fail immediately on permanent API errors rather than waiting for a polling timeout.
• Retry only safe reads and idempotent updates, upserts, state changes, and deletes.
• Apply the same retry policy to cleanup operations.
• Add an explicit management-request timeout.
• Do not blindly retry resource-creation POSTs after ambiguous transport or server failures. Reconcile by deterministic run tags/name first so retries cannot create duplicate resources.
• Add mock coverage for transient-success, terminal-error, timeout, and cleanup cases.

2. Select suites by affected domain

Keep one workflow with a small changed-path classifier rather than duplicating the complete secret-bearing job across several workflows.

Changed domain	Live suites
Services	Service, ClickPipe CDC
Postgres	Postgres, ClickPipe CDC
ClickPipes	ClickPipe CDC, smoke
Organizations, activity, members, RBAC	Organization
API keys	Service, Organization
ClickStack, UDFs, backups	None until a corresponding live suite exists
CLI Cloud modules	None; regular CLI CI owns these
Shared client, error, models, test support, manifests	All

Scheduled and explicitly requested full runs should select every suite. Unknown new API source paths should fail closed by selecting all suites.

Selected suites can remain sequential in one job to reuse compilation and avoid creating another traffic burst. Each selected step should still run if an earlier selected suite fails, followed by one final result step that reports the aggregate failure.

3. Make live CI on-demand

The leading option is a maintainer-applied label such as can be tested or run-cloud-integration.

Recommended semantics:

• The expensive live workflow responds to the pull_request labeled event, not every synchronize event.
• Applying the full-suite label is a one-shot trigger for the PR head SHA at that moment.
• A maintainer can apply it only to the top PR of a stack. Because that branch contains every lower stack commit, a full run validates the combined stack.
• New commits do not automatically launch another live run. Remove and reapply the label when another run is wanted.
• The workflow may remove the trigger label after enqueueing so it behaves like a button, although that requires narrowly scoped pull-requests: write permission.
• The full-suite label must override changed-path selection. The top PR diff is relative to its immediate stacked base and does not list changes made by lower PRs, even though those commits are present in the tested checkout.
• Keep the weekday scheduled run as a full sweep against main.

GitHub label permissions provide the intended approval boundary: users need triage/write access to apply repository labels. Fork PRs still do not receive repository secrets under pull_request; do not switch to pull_request_target and execute untrusted PR code. A maintainer-owned mirror branch is the safe route when a fork needs live validation.

Other trigger options

Manual workflow dispatch

Add workflow_dispatch inputs for PR/ref and scope (all, affected, service, postgres, org, or clickpipes). This is secure and explicit for maintainers, but less discoverable from the PR page and requires care to ensure the exact intended SHA is tested.

PR comment command

A command such as /run-cloud-ci all is convenient and can support scopes, but authorization and secret handling are more complex. An issue_comment/pull_request_target workflow must only validate the commenter and dispatch a separate trusted workflow; it must never directly check out and execute PR code with secrets.

Persistent approval label

Treat the label as approval and continue running on every later push while it remains attached. This resembles the main ClickHouse repository model, but it is less suitable here because repeated pushes to a labeled PR can recreate the same load. One-shot label semantics are preferable for this repository.

Environment approval

Trigger workflows automatically but hold the live job behind a protected GitHub environment. This prevents unapproved API access but leaves many pending workflow runs and becomes awkward during stack refreshes. It is not preferred.

Ready-for-review or merge-ready trigger

Run once when a PR leaves draft or receives approval. This is lightweight but less explicit, and review changes do not always correspond to the exact moment a live Cloud run is useful. It could complement, but should not replace, a manual trigger.

Suggested rollout

1. Add retry/backoff, request timeouts, and cleanup hardening.
2. Remove CLI-only live triggers and add domain-aware suite selection.
3. Add a one-shot maintainer label for a full run, retaining manual and scheduled full runs.
4. Optionally add scoped labels or workflow_dispatch inputs for targeted reruns.
5. Observe retry logs, API status distribution, run frequency, and leaked-resource counts before changing concurrency further.

Acceptance criteria

• A CLI-only, ClickStack-only, UDF-only, or backup-only PR update does not launch live Cloud tests.
• A domain-targeted invocation runs only the mapped suites.
• Applying the full-suite label to the top PR of a stack tests that exact head SHA and runs all suites once.
• Subsequent pushes do not automatically rerun the one-shot labeled workflow.
• Transient 429 and retryable 5xx responses show bounded backoff and can recover within the same test run.
• Cleanup uses the same transient retry policy and reports any final failure without losing the original test error.
• Scheduled runs against main continue to exercise the full suite.

[sdairs]

Confirmed: this is possible with gh stack modify. We can insert two CI-fix branches between #389 and #390, implement them lower-to-upper, cascade-rebase the branches above, then run gh stack submit once. Existing PR numbers stay intact; #390 and everything above it will be rebased and CI will rerun. Before modifying, run gh stack rebase --no-trunk because #391 currently needs rebasing.

[sdairs]

Repo-wide serialization is possible directly in GitHub Actions. Change the cloud integration concurrency from per-ref to:

concurrency:
  group: cloud-integration
  queue: max
  cancel-in-progress: false

This runs one cloud integration workflow at a time and queues up to 100 additional PR, manual, or scheduled runs, allowing them to waterfall unattended instead of colliding. We should make this change.

[sdairs]

Investigation findings and revised proposal

I reviewed the current workflows, integration-test harness, and recent Cloud Integration runs. The direction in this issue is right. I recommend tightening several design decisions and changing the rollout order so we contain the immediate failure mode before making complex retry changes.

What recent runs show

The August 13 stack refresh created 22 runs across 13 branches in about five minutes, with effective live-job concurrency reaching 13. All 22 runs failed. The failures included 429, query-endpoint 503, Query API 404, and Postgres 500 responses. Cleanup also encountered 429 responses.

Concurrency is not the only problem. Isolated scheduled runs have also failed with Postgres and ClickPipe 500 responses. We need both admission control and operation-specific resilience.

Recommended first change: contain live execution

Make the workflow on-demand and globally serialized before changing API behavior:

1. Trigger PR runs only from a maintainer-applied run-cloud-integration label.
2. Keep scheduled full runs and scoped workflow_dispatch runs.
3. Put every secret-bearing live job in one global concurrency group with cancel-in-progress: false.
4. Check out the exact PR head SHA captured by the label event.
5. Protect the cloud-integration environment with required reviewers and prevention of self-approval.
6. Keep ordinary build, unit, clippy, and formatting workflows automatic.

The label controls admission. Global concurrency controls execution. Both are required because maintainers can label multiple PRs, and scheduled or manual runs can otherwise overlap.

This should be first because it immediately removes the load multiplier without changing API semantics. It cannot duplicate a create, replay a restart, or alter cleanup behavior. It is also small and reversible.

One caveat needs an explicit decision: GitHub concurrency is not a durable queue. It permits one running and one pending run, and a newer pending run can replace the older pending run. If latest-pending-wins is acceptable, a global group is sufficient. If every requested run must execute, we need an external lock or queue.

Trigger and SHA details

Trigger-level paths must be removed for labeled runs. GitHub combines the event and path filters, so a full-run label can otherwise fail to trigger when the immediate PR diff has no matching path. Use pull_request: types: [labeled] and gate the job on the exact label name.

The current unqualified actions/checkout tests the synthetic PR merge ref. If the acceptance criterion is the exact head SHA at label time, use github.event.pull_request.head.sha explicitly. Also pass the tested SHA to the test harness: GITHUB_SHA remains the event merge SHA even when checkout uses another ref. Log the PR head, base, event SHA, and checked-out SHA.

A full run on the top stack PR validates that combined code snapshot. It does not produce successful checks for lower PRs and does not validate later merge or squash rewrites.

An on-demand workflow should remain informational. It cannot be a normal required check because later pushes receive no new run. If it becomes required later, add an always-running lightweight gate that verifies a successful live run exists for the current SHA.

I recommend leaving the trigger label attached initially. The labeled event is already one-shot, and manual removal plus reapplication supports another run without granting pull-requests: write or creating label-consumption races.

Simplify suite selection initially

The affected-domain classifier currently has no concrete caller: the proposed PR label runs all suites, and schedules also run all suites. The simpler first version is:

• run-cloud-integration label: all mandatory suites.
• Schedule: all mandatory suites.
• workflow_dispatch: explicit all, service, postgres, organization, or clickpipes scope.

This already supports targeted reruns. Add an affected classifier later only if it provides enough value to justify its maintenance cost.

If we retain affected, its semantics need tests and broader mappings. organizations.rs is exercised by both the service and organization suites. Shared models, conversions, client/error code, test support, manifests, workflow changes, deleted files, and renames also need classification. Unknown paths should select all. Known domains without a live suite should map explicitly to none. Suite selection should be described as relevance, not endpoint coverage: selecting the ClickPipes suite does not mean every ClickPipes endpoint is exercised.

Define full precisely. Today the smoke suite is skipped when CLICKHOUSE_CLOUD_TEST_CLICKPIPE_SERVICE_ID is unset. Full must mean either four mandatory suites plus optional smoke, or the fixture must become mandatory.

Retry policy must be operation-specific

Retry safety is semantic, not based only on the HTTP verb.

Operation	Recommended behavior
Management GET/list	Retry 429, selected 5xx, and timeout/connect/reset errors within one deadline
Absolute PUT or singleton upsert	Reconcile current state after an ambiguous result, then retry if still needed
DELETE during cleanup	Treat 404 as success; GET after an ambiguous failure before another DELETE
Collection create POST	Never blindly retry; discover and register matching candidates first
Add/remove collection PATCH	Read current state and determine whether the mutation is still needed
Start/stop	Reconcile desired state before replay
Restart or generated password rotation	Never blindly replay
Query API SELECT	Retry only when the caller explicitly declares it replay-safe
Query API INSERT, DDL, or other writes	Never replay after an ambiguous result

The retry design should specify exact statuses, maximum attempts, elapsed deadline, base delay, delay cap, jitter algorithm, and request timeout. 500, 502, 503, and 504 are reasonable initial retryable server statuses; not every 5xx should be treated alike.

poll_until currently absorbs every error until timeout. Replace its Result<Option<T>> behavior with explicit ready, pending, retryable-error, and terminal-error outcomes. Authentication, validation, malformed responses, and terminal resource states should fail immediately.

Use the existing custom HTTP-client constructor to set test-specific connect and total request timeouts. Automatic mutation retries should remain in test support rather than becoming hidden production-client behavior. The public error type may need richer metadata because it currently discards Retry-After, request IDs, and response headers.

Backoff tests should use injectable time, sleep, and randomness so they do not sleep in real time or assert nondeterministic delays.

Ambiguous creates need credential-aware reconciliation

Deterministic names and tags prevent duplicate creates, but they do not always let a test continue. ClickHouse service, Postgres, and API-key creation responses contain credentials that cannot necessarily be recovered from GET/list endpoints.

After an ambiguous create result:

1. Search for exact ownership markers within a bounded consistency window.
2. Register every matching resource for cleanup before returning.
3. Adopt exactly one match only when all required information can be recovered.
4. If credentials cannot be recovered, delete the match and fail without issuing another create first.
5. If multiple matches exist, register and delete all of them and fail loudly.

The current run identity also needs separation between a logical run label and a unique attempt ID. Reapplying a label to the same SHA currently reuses resource names because the human label takes precedence over GITHUB_RUN_ID. Use the logical PR/nightly label for grouping and a run/attempt/suite token for names and exact ownership.

Cleanup requires more than retries

Current cleanup is not guaranteed in several cases:

• Assertions and panics can unwind past cleanup.
• Some provisioning paths can create a resource before a cleanup owner exists.
• Parallel provisioning can process one failed result before registering the other successful resource.
• ClickPipe driver cleanup errors are discarded when the test already failed.
• Smoke cleanup failures are logged without failing the test.
• Runner cancellation or timeout can never execute in-process cleanup.

Make each resource-owning lifecycle panic-safe, process and register every successful parallel result before returning an error, and aggregate the original test failure with all Cloud and external cleanup failures. Keep cleanup registrations until absence is confirmed.

Add a scheduled stale-resource janitor using ownership tags, attempt IDs, and a conservative age threshold. In-process cleanup cannot cover runner loss or forced cancellation.

Use one deadline hierarchy rather than nesting full retry and polling budgets. Reserve cleanup time before suite and job deadlines. A workflow timeout should be the last safety net because killing the process bypasses cleanup.

Ordinary CI gap

API changes run test-cloud-api.yml but do not trigger test-cli.yml, even though the CLI depends on the local API crate. An API model or method signature change can therefore break the CLI without compiling it in that PR workflow. Add API source and manifest paths to Test CLI, or add an always-on workspace compile check.

Revised rollout

1. Add the one-shot full label, scoped manual dispatch, exact-SHA checkout, protected environment, and global serialization.
2. Add Rust caching, explicit request/suite/job deadlines, stable aggregate results, and structured job summaries.
3. Fix unique run identity, panic-safe cleanup, provisioning ownership, and stale-resource cleanup.
4. Add operation-specific retries and create reconciliation with deterministic mock coverage.
5. Add affected-path classification only if targeted manual scopes prove insufficient.

This order first stops the current traffic burst, then establishes observability and safety, and only then adds retries. Generic retries added before reliable identity, reconciliation, and cleanup could amplify duplicate mutations and leaks.

Suggested additional acceptance criteria:

• At most one live job mutates the shared organization at a time.
• Fork and Dependabot PRs never enter the secret-bearing job.
• Logs identify the exact tested SHA and unique attempt ID.
• Ambiguous credential-bearing creates cannot produce a second create before reconciliation.
• Panics still attempt cleanup and preserve both failures.
• Cancellation leaks are discoverable by the stale-resource janitor.
• API changes compile and test the CLI in ordinary CI.

[sdairs]

Consolidated direction

This consolidates the issue body and follow-up research into one target design. It also resolves the conflicting points in the earlier comments.

Operating model

Invocation	Selection	Tested revision
PR label run-cloud-integration	Suites affected by that PR's immediate diff	Head SHA captured by the labeled event
workflow_dispatch	Explicit all, service, postgres, organization, or clickpipes scope	SHA selected when the workflow is dispatched
Weekday schedule	All suites	Scheduled main SHA

• The PR label is one-shot because only pull_request: types: [labeled] triggers it. Leave the label attached; remove and reapply it to request another run.
• A later push does not trigger another live run.
• A full stack validation is a manual all dispatch against the top stack branch. This tests the combined snapshot. The affected label on a stacked PR classifies only that PR's immediate base-to-head diff.
• all means the four mandatory service, Postgres, organization, and Postgres CDC suites, plus ClickPipe smoke when CLICKHOUSE_CLOUD_TEST_CLICKPIPE_SERVICE_ID is configured.
• Scoped clickpipes means Postgres CDC plus optional smoke. The other manual scopes run their one corresponding suite.
• The live result remains informational, not a required check. A one-shot result cannot safely be required after the PR head changes.
• Ordinary build, unit, clippy, and formatting CI remains automatic.

Admission and serialization

Use a secretless planning job followed by the environment-bearing live job.

• The planning job resolves scope, classifies changed paths, records the exact SHA, and exposes the selected suites. It receives no environment and no Cloud secrets.
• The live job starts only when at least one suite is selected.
• PR live jobs require a same-repository head branch and exclude Dependabot. Fork PRs use a maintainer-owned mirror branch when live validation is needed.
• Keep pull_request; do not use pull_request_target to execute PR code.
• The maintainer-controlled label is the approval boundary. Do not add required environment reviewers: that would duplicate approval and block unattended scheduled runs unless environments and secrets were split.
• Keep the environment as the secret container with contents: read permissions. Do not grant pull-requests: write or auto-remove the label.
• Put every shared-organization mutator, including the future janitor, in concurrency.group: cloud-integration with queue: max and cancel-in-progress: false.

Current GitHub Actions supports queue: max, retaining up to 100 pending runs. This resolves the earlier one-pending-run caveat; no external queue is needed. Ordering is not guaranteed, but execution is serialized.

Affected-suite classifier

Classify the union of every changed path. Use the captured base and head SHAs, not the PR's current state when a queued job eventually starts.

Changed area	Suites
Shared client, error, serde, model/conversion facades, shared models/conversions, common live-test support, manifests, lockfile, Rust config, or Cloud workflow/classifier	All
Service client/models/conversions	Service, ClickPipes
Postgres client/models/conversions	Postgres, ClickPipes
ClickPipes client/models and selected CDC/smoke support	ClickPipes
Organizations, BYOC organization fields, and organization private endpoints	Service, Organization
API keys	Service, Organization
Activity, invitations, members, and RBAC	Organization
The individual live integration test file	Its corresponding suite
ClickStack, UDFs, backups, quotas, SCIM, metadata, unselected ClickPipes E2E files, unit-only API tests, OpenAPI snapshot, docs, or CLI code	None
Unknown new API source/test path or classifier failure	All

The mapping expresses relevance, not endpoint coverage. In particular, selecting ClickPipes does not claim that all ClickPipes endpoints are covered.

Classifier requirements:

• Parse additions, modifications, deletions, copies, and renames; classify both old and new rename paths.
• Union suites across files; all dominates.
• Fail closed to all for an unknown API source/test path, malformed diff, missing SHA, or classifier failure.
• Treat a known no-suite path explicitly rather than through broad globs that could hide a future domain.
• Add table-driven mapping tests and an inventory test that requires every current API source file to have an explicit mapping.
• A no-suite result succeeds in the planning job and never enters the secret-bearing job.

Workflow execution and reporting

• Check out the exact captured head SHA for labeled PRs and the exact dispatch/schedule SHA for other events.
• Log event SHA, base SHA, head SHA, checked-out SHA, logical run label, GitHub run/attempt ID, requested scope, and selected suites.
• Run selected suites sequentially to avoid another traffic burst.
• Run every selected suite even if an earlier suite fails, then use one final aggregate step to fail the job.
• Write a job summary with each selected suite's result and whether smoke was run or unavailable.
• Add the existing pinned Rust cache action to avoid recompiling from scratch on every live run.
• Do not add a hard job timeout until suite deadlines reserve cleanup time and the stale-resource janitor exists. Forced job termination bypasses in-process cleanup.

Run identity and ownership

Separate human grouping from exact ownership.

• Logical label: pr-N, nightly, or manual.
• Unique attempt ID: GitHub run ID plus run attempt, with suite identity where names can collide.
• Tested revision: full checked-out SHA, with a short SHA only for human-readable names.
• Persist managed_by, logical label, unique attempt ID, suite, and tested SHA wherever the resource supports tags.
• Use deterministic names containing a compact unique-attempt token for untaggable resources.

Register ownership as soon as a resource ID is known, before readiness checks, credential assertions, or follow-up configuration. Process every parallel provisioning result and register all successes before returning any failure.

Cleanup contract

Cleanup correctness comes before mutation retries.

• Catch panics around each resource-owning lifecycle, run cleanup, then preserve the panic together with cleanup failures.
• Aggregate the original test failure and every Cloud or external cleanup failure.
• Keep cleanup entries registered until absence or restoration is confirmed.
• Fix smoke and E2E paths that currently log or discard cleanup failures.
• Treat delete 404 as success. After an ambiguous delete result, reconcile existence before another delete.
• Restore shared state and delete children before parents.
• Use one deadline hierarchy; do not nest full polling and retry budgets. Each suite must reserve a cleanup budget before its execution deadline.

Retry and polling policy

Retries remain explicit test-harness behavior, not hidden production-client behavior.

Initial defaults to pin in tests and tune from scheduled-run evidence:

• Test-only management client connect timeout: 10 seconds.
• Test-only management request timeout: 60 seconds or the remaining operation deadline, whichever is shorter.
• Direct-operation retry budget: at most 8 attempts and 2 minutes elapsed.
• Backoff: base 1 second, cap 30 seconds, exponential full jitter.
• Honor Retry-After when present and within the remaining deadline.
• Retryable server statuses: 429, 500, 502, 503, and 504; do not classify every 5xx as retryable.
• Retryable transport failures: timeout, connect, and reset/closed-connection failures when the operation is replay-safe.

Replace poll_until's Result<Option<T>> contract with explicit ready, pending, retryable-error, and terminal-error outcomes. Successful pending checks use the normal poll cadence; transient failures use backoff; terminal resource states, authentication, validation, and malformed responses fail immediately.

Operation rules:

Operation	Behavior
Management GET/list	Retry within one deadline
Explicitly replay-safe Query API SELECT	Retry only when the caller opts in
Absolute PUT or singleton upsert	Read current state after ambiguity, then retry only if still needed
Add/remove collection PATCH	Read current state and recompute the remaining mutation
Start/stop	Reconcile desired state before replay
Cleanup DELETE	Treat absent as success; reconcile after ambiguity before replay
Restart or generated-password rotation	Never blindly replay
Query API INSERT, DDL, or other writes	Never replay after ambiguity
Collection create POST	Never blindly retry

Preserve response headers needed for Retry-After and request IDs in the public error metadata. Backoff tests must inject time, sleeping, and randomness so they are deterministic and do not sleep.

Ambiguous collection creates

A credential-bearing create cannot safely be treated like an idempotent mutation.

• Give every create deterministic exact ownership markers before sending it.
• After a timeout, transport failure, or retryable server response, search for exact matches within a bounded consistency window before any further action.
• Register every match for cleanup before returning.
• Adopt exactly one match only when all information required to continue is recoverable.
• If credentials cannot be recovered, delete the match and fail. Do not issue another create first.
• If multiple matches exist, register and delete all of them and fail loudly.
• If no match appears within the consistency window, fail rather than risk a delayed duplicate.

Stale-resource janitor

In-process cleanup cannot handle runner cancellation or forced termination.

• Add a scheduled janitor that shares the global concurrency group.
• Start in report-only mode, then enable deletion after ownership markers and age calculations have been observed on scheduled runs.
• Delete only resources with exact test ownership markers and a conservative age threshold greater than the maximum live-run duration.
• Clean children before parents and aggregate all failures.
• Report resources that cannot be proven safe to delete rather than using a broad name prefix.

Ordered child issues

1. Make live Cloud CI on-demand, affected-aware, scoped, and globally serialized. Add the planning/live job split, one-shot label, classifier and tests, scoped dispatch, scheduled full run, exact-SHA checkout, same-repo guards, and global queued concurrency. This is the immediate containment change.
2. Compile the CLI consumer when the Cloud API changes. Add Cloud API source/manifests and the workspace manifest to test-cli.yml triggers. This is independent and should land immediately.
3. Run and report all selected live suites. Add Rust caching, sequential continue-on-error suite execution, final aggregate failure, SHA/scope logging, and the job summary. Depends on issue 1.
4. Give every live test attempt a unique ownership identity. Separate logical labels from run/attempt/suite identity and standardize names/tags. Depends on issue 1.
5. Make live provisioning and cleanup panic-safe. Register ownership earlier, collect parallel results safely, preserve test plus cleanup failures, retain failed registrations, and fix smoke/E2E cleanup gaps. Depends on issue 4.
6. Add bounded management requests and typed retry/polling primitives. Preserve retry metadata, add test-only request limits, implement explicit polling outcomes and deterministic backoff tests. Can proceed alongside issues 4 and 5, but must not introduce broad mutation retries yet.
7. Apply operation-safe retries and reconciliation. Migrate reads, polling, cleanup deletes, and explicitly reconciled updates/state changes. Depends on issues 5 and 6.
8. Reconcile ambiguous collection creates. Add exact-match discovery, adoption rules, credential-aware failure behavior, and deterministic duplicate-prevention tests. Depends on issues 4, 5, and 6; it can proceed alongside issue 7.
9. Add the stale-resource janitor and top-level deadline hierarchy. Add report-only then active stale cleanup, reserve cleanup time in suite budgets, and only then add a final workflow timeout. Depends on issues 4, 5, 7, and 8.

The critical dependency is identity and cleanup before mutation retries. Generic retries added earlier could turn transient failures into duplicate credential-bearing resources and larger leaks.

Parent acceptance criteria

• PR pushes never launch live Cloud tests automatically.
• Applying run-cloud-integration once selects the affected suites and tests the exact head SHA captured at label time.
• Reapplying the label is required for another PR run; later pushes do not reuse the old approval.
• A manual all run against the top stack branch tests that exact combined snapshot.
• At most one live job or janitor mutates the shared organization at a time, while additional requested runs remain queued.
• Fork and Dependabot PRs never enter the secret-bearing job.
• Known no-suite changes do not enter the live job; unknown API source/test changes select all.
• Every selected suite runs and the final result preserves all suite failures.
• Scheduled main runs continue to execute all mandatory suites and optional smoke.
• API changes compile and test the CLI in ordinary CI.
• Transient retryable reads and safe mutations can recover with bounded, visible backoff; permanent errors fail immediately.
• Ambiguous credential-bearing creates cannot cause a second create and all discovered matches are registered for cleanup.
• Panics still attempt cleanup and preserve both the original and cleanup failures.
• Cancellation leaks are discoverable and, after the observation phase, removable by the janitor.

[sdairs]

Implementation decision

This supersedes the earlier rollout ordering in the issue body and comments for the immediate CI work.

We will separate traffic containment from suite targeting and defer affected-path classification until the API module tree is stable:

1. Make live Cloud integration CI on-demand and globally serialized #404 makes live PR CI one-shot and maintainer-triggered, globally serializes every live run, and pins PR execution to the captured head SHA.
2. Add scoped manual Cloud integration runs #405 adds explicit manual all, service, postgres, organization, and clickpipes scopes without introducing path inference.
3. Select live Cloud integration suites from affected API domains #406 adds affected-domain classification after Split Cloud API client operations and conversions by domain #391 through Extract remaining ClickStack API models and finalize the library split #396 land. It is intentionally blocked until then.

#404 and #405 should be inserted together between #389 and #390 so rebasing the existing stack no longer launches live Cloud tests for every PR. Implement them lower-to-upper, cascade-rebase the stack, and submit it once.

Do not put #406 below the modularization work. At #389, API operations and models are still concentrated in monolithic files, so the classifier would either select all for most changes or encode paths that do not exist yet. After modularization, domain-only changes have stable, explicit paths and an inventory test can require every API source file to be mapped.

For the current stack, run ordinary CI automatically and perform one manual all run against the top branch to validate the combined snapshot. Later, the run-cloud-integration label can use #406 for the immediate PR diff; full stacked validation remains a manual all run because inherited lower-PR changes are absent from the top PR immediate diff.

Retry, timeout, identity, cleanup, and stale-resource work remains valid follow-up under this parent, but it should not block these three workflow changes and should not be mixed into their implementation PRs.