ci: collect cluster diagnostics for failing component tests

[danish9039]

Summary of Changes

Seven component workflows ended without capturing anything when a job failed, so
a red run left only the console output of the step that timed out.

• Add a Collect Logs on Failure step and a diagnostic-log artifact to the
  dashboard, Dex with OAuth2 Proxy, Katib, notebooks authorization, pipeline run
  from notebook, volumes web application, and workspaces pipeline workflows.
• Follow the pattern already used by the KServe, Ray, Trainer, Istio validation,
  and full integration workflows.
• Collect the shared part through tests/collect_cluster_diagnostics.sh instead
  of repeating it in every workflow.

Why a shared script

Five workflows already carry their own copy of this block. Inlining it seven
more times would leave thirteen copies to keep in step. The shared script makes
this change 89 lines across the workflows rather than roughly 168, while
collecting strictly more than the existing copies do.

What is collected beyond the existing pattern

The current blocks record cluster resources, events, pod descriptions and pod
logs. They cannot answer why a pod that was ready earlier stopped being ready.

• Node descriptions report memory, disk and process identifier pressure, which
  distinguishes a stalled workload from a runner that ran out of capacity and
  evicted healthy pods.
• Runner memory, disk and load report the same limit from outside the cluster.
• Persistent volume claims explain readiness timeouts that cascade from unbound
  storage.
• Wide pod output carries restart counts.
• Previous container logs explain a container that already died.
• The kube-system and local-path-storage namespaces are always collected,
  because a stalled control plane or storage provisioner surfaces as an
  unrelated component timing out.

Safety

Every command is best effort and the script always exits zero, so collecting
diagnostics can never mask the failure that triggered it. This was verified by
running the script with no cluster reachable.

There is no change to any installation path, so no test behaviour changes.

Verification

• pre-commit on every changed file: yamllint, shellcheck and hygiene hooks pass.
• All seven workflows parse and contain both new steps.
• The script runs without a reachable cluster and exits zero.
• upload-artifact is pinned to the same commit already used elsewhere in the
  repository.

Contributor Checklist

• All commits are signed off to satisfy the DCO check.

Related Issues

Tracking issue for the failure these diagnostics were built to capture:
#3563

[github-actions]

Welcome to the Kubeflow Community Distribution Repository

Thanks for opening your first PR. Your contribution means a lot to the Kubeflow community.

Before making more PRs:
Please ensure your PR follows our Contributing Guide.
Please also be aware that many components are synchronized from upstream via the scripts in /scripts.
So in some cases you have to fix the problem in the upstream repositories first, but you can use a PR against kubeflow/community-distribution to test the platform integration.

Community Resources:

• CNCF Slack channels: https://www.kubeflow.org/docs/about/community/#kubeflow-slack-channels
• Community meetings: https://www.kubeflow.org/docs/about/community/

Thanks again for helping to improve Kubeflow.

[Copilot]

Pull request overview

Centralizes failure diagnostics for seven component workflows, capturing cluster, node, storage, host, and container evidence.

Changes:

• Adds a shared diagnostic collection script.
• Uploads diagnostic artifacts for seven workflows.
• Adds component-specific resource snapshots and pressure sampling.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
tests/collect_cluster_diagnostics.sh	Implements shared cluster diagnostics.
.github/workflows/dashboard_test.yaml	Adds dashboard failure diagnostics.
.github/workflows/dex_oauth2-proxy_test.yaml	Adds identity-service failure diagnostics.
.github/workflows/katib_test.yaml	Adds Katib diagnostics and resource snapshots.
.github/workflows/notebooks_api_authorization_test.yaml	Adds notebook authorization diagnostics.
.github/workflows/pipeline_run_from_notebook.yaml	Adds pipeline and notebook diagnostics.
.github/workflows/volumes_web_application_test.yaml	Adds volume application diagnostics.
.github/workflows/workspaces_pipeline_run_test.yaml	Adds workspace diagnostics and pressure sampling.

Comments suppressed due to low confidence (1)

tests/collect_cluster_diagnostics.sh:45

• This pipeline also converts a failed node query into an empty value, which the output formatter reports as none. That makes an unreachable API server indistinguishable from a healthy node with no pressure conditions; retain the kubectl status and reserve none for a successful query with no matching conditions.

        node_pressure=$(kubectl get nodes \
            -o jsonpath='{range .items[*]}{range .status.conditions[?(@.status=="True")]}{.type} {end}{end}' \
            2>/dev/null | tr ' ' '\n' | grep -E 'Pressure' | sort -u | tr '\n' ',' | sed 's/,$//')

[danish9039]

Fixed: previous logs are now collected per container, selected by restartCount>0, so a stable sidecar no longer causes the restarted application container's prior log to be discarded.

[danish9039]

Removed, since the sampler was temporary instrumentation for diagnosing #3563 and its cost during installation is not justified now that the cause is known.

[danish9039]

Fixed: this workflow now uses failure() || cancelled() like the other six, so no green run uploads an artifact.

[danish9039]

Not adding these paths, because the diagnostic step runs only on failure, so a run triggered by editing this script would pass without ever executing it and would give no coverage while consuming seven end to end jobs. shellcheck already validates the script through pre-commit, and a smoke test asserting it exits zero without a cluster would be the appropriate coverage if more is wanted.

[danish9039]

Resolved by removing the script, so there is no shared implementation left to reference from the trigger paths.

[juliusvonkohout]

please remove this

[danish9039]

Removed, since this sampler was temporary instrumentation used to find the cause of the pipelines readiness flake, which is now recorded as API server degradation in #3563.

[juliusvonkohout]

do we really need this ?

[danish9039]

No, removed together with the sampler step: it existed only to capture the pressure time series while diagnosing #3563, and the failure only diagnostics that remain are what produced that evidence.

[danish9039]

The whole script is now removed. Reviewing what the evidence actually required, every finding that identified the cause in #3563 came from cluster events, pod descriptions and pod logs, which the block already used by the full integration, KServe, Ray and Trainer workflows collects. These seven workflows now use that same block, so the pull request only adds the coverage that was missing and introduces no new pattern.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (3)

tests/collect_cluster_diagnostics.sh:78

• This query examines only status.containerStatuses, so restarted initialization containers are omitted. Pipelines installed by the consuming workflows contain wait-for-* initialization containers, for example applications/pipeline/upstream/base/pipeline/ml-pipeline-apiserver-deployment.yaml:24, and the prior instance can explain why a pod never becomes ready. Include status.initContainerStatuses in the selection.

            -o jsonpath='{.status.containerStatuses[?(@.restartCount>0)].name}' \

tests/collect_cluster_diagnostics.sh:29

• This comment is inaccurate: a pod can lose readiness without restarting, and kubectl describe already reports restart counts and prior container state. Describe this output as a cluster-wide summary instead of asserting that a readiness transition appears as a restart.

# Pod restart counts and placement. A pod that was ready earlier and is not
# ready now shows up here as a restart, which a describe of the current state
# alone does not reveal.

tests/collect_cluster_diagnostics.sh:81

• previous_log stores a file path rather than log content, so the name is ambiguous and conflicts with the expressive naming requirement in AGENTS.md:7. Rename it to identify the value as the previous container log file path.

            previous_log="$OUTPUT_DIRECTORY/$namespace-$pod_name-$container_name-previous.txt"

[danish9039]

Fixed: every call now goes through a wrapper that sets --request-timeout and an outer per-call deadline, and collection stops once an overall budget is spent, so a degraded API server can no longer hang the step and cost us the artifact upload. Verified with a stubbed kubectl that never returns, where collection now ends in seconds instead of running to the job limit.

[danish9039]

Resolved by removing the script. These workflows now use the same inline block as the four existing consumers on master. The unbounded request concern applies equally to that existing block, so it is better handled once for all workflows in a separate change than introduced here.

[danish9039]

Reopening this: the concern is correct and now fixed here rather than deferred. kubectl options documents --request-timeout='0' as not timing out requests at all, and six of these seven workflows set no job level timeout, so the default of three hundred and sixty minutes would apply to a hang. The collection step now carries timeout-minutes: 5. A step that exceeds its own timeout fails rather than cancelling the job, so the upload still runs and partial evidence is preserved. The four workflows that already carry this block on master have the same gap, and are left for a follow up because two of them are also modified by #3515.

[danish9039]

Extended to every workflow rather than deferred. All eleven diagnostic collection steps in the repository now carry timeout-minutes: 5, including the four that already had this block before this pull request. The overlap with #3515 on kserve_test.yaml and full_kubeflow_integration_test.yaml is a three line addition in each and is straightforward to resolve whichever merges first.

[danish9039]

Removed, that marker was left over from reproducing the failure and should not have been committed.

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 12 comments.

Comments suppressed due to low confidence (7)

.github/workflows/workspaces_pipeline_run_test.yaml:89

• These workloads use Istio sidecars, but kubectl logs without a container selector returns only the default container. The sidecar connection failures highlighted by issue 3563 are therefore omitted from the artifact. Collect every container and prefix each line so interleaved output remains attributable.

                kubectl logs -n $namespace $pod --tail=100 > logs/$namespace-$pod.txt 2>&1 || true

.github/workflows/volumes_web_application_test.yaml:88

• These workloads use Istio sidecars, but kubectl logs without a container selector returns only the default container. The sidecar connection failures highlighted by issue 3563 are therefore omitted from the artifact. Collect every container and prefix each line so interleaved output remains attributable.

                kubectl logs -n $namespace $pod --tail=100 > logs/$namespace-$pod.txt 2>&1 || true

.github/workflows/pipeline_run_from_notebook.yaml:97

• These workloads use Istio sidecars, but kubectl logs without a container selector returns only the default container. The sidecar connection failures highlighted by issue 3563 are therefore omitted from the artifact. Collect every container and prefix each line so interleaved output remains attributable.

                kubectl logs -n $namespace $pod --tail=100 > logs/$namespace-$pod.txt 2>&1 || true

.github/workflows/notebooks_api_authorization_test.yaml:108

• These workloads use Istio sidecars, but kubectl logs without a container selector returns only the default container. The sidecar connection failures highlighted by issue 3563 are therefore omitted from the artifact. Collect every container and prefix each line so interleaved output remains attributable.

                kubectl logs -n $namespace $pod --tail=100 > logs/$namespace-$pod.txt 2>&1 || true

.github/workflows/katib_test.yaml:94

• These workloads use Istio sidecars, but kubectl logs without a container selector returns only the default container. The sidecar connection failures highlighted by issue 3563 are therefore omitted from the artifact. Collect every container and prefix each line so interleaved output remains attributable.

                kubectl logs -n $namespace $pod --tail=100 > logs/$namespace-$pod.txt 2>&1 || true

.github/workflows/dex_oauth2-proxy_test.yaml:87

• These workloads use Istio sidecars, but kubectl logs without a container selector returns only the default container. The sidecar connection failures highlighted by issue 3563 are therefore omitted from the artifact. Collect every container and prefix each line so interleaved output remains attributable.

                kubectl logs -n $namespace $pod --tail=100 > logs/$namespace-$pod.txt 2>&1 || true

.github/workflows/dashboard_test.yaml:52

• These workloads use Istio sidecars, but kubectl logs without a container selector returns only the default container. The sidecar connection failures highlighted by issue 3563 are therefore omitted from the artifact. Collect every container and prefix each line so interleaved output remains attributable.

                kubectl logs -n $namespace $pod --tail=100 > logs/$namespace-$pod.txt 2>&1 || true

[juliusvonkohout]

/lgtm
/approve

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: juliusvonkohout

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [juliusvonkohout]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment