[AIRE] CI failure: Jenkins agent went offline during build_docker_image in run 32313929250 (PR #31)

[svc-bcs-agent]

Bug Report

Bug Groups in This Run

group	slug	status	one-line root cause
g1	jenkins-agent-offline-docker-build	needs-investigation	The Jenkins Kubernetes agent pod lost its websocket connection to the controller during the build_docker_image stage, aborting the pipeline after all tests had already passed

Related PR: none (no auto-fixable bug — the failure is infrastructure-side and has no in-repo remedy)

Description

g1: jenkins-agent-offline-docker-build

Problem Statement

Blossom-CI run 32313929250 (PR #31) reports conclusion: success on GitHub, but the mirrored internal Jenkins build in the "Upload log" job ended in failure. Everything that validates the contributor's change succeeded:

• dependency installation completed (Successfully installed ... nvidia-tao-core-7.0.0 ...),
• the unit-test suite passed in full — 448 passed, 23 skipped, 112 warnings in 954.60s (0:15:54),
• coverage was written (coverage.xml, 475973 bytes) and the SonarQube analysis on branch pr_31 reported analysis completed successfully,
• cleanup_scratch_space and docker_daemon_setup both completed, and docker login to nvcr.io returned Login Succeeded.

The pipeline then entered the build_docker_image stage and printed Pre-flattening base image <redacted> -> tao-flat-base-amd64:local (stage base-amd64). At the very next sh step the Jenkins controller lost contact with the build agent:

Cannot contact tao-ds-oss-develop-50-2njfx-xnrzg-2x92t: hudson.remoting.RequestAbortedException: java.nio.channels.ClosedChannelException
...
ERROR: Execution failed
java.nio.channels.ClosedChannelException
	at jenkins.agents.WebSocketAgents$Session.closed(WebSocketAgents.java:160)
...
org.jenkinsci.plugins.workflow.support.steps.AgentOfflineException: Unable to create live FilePath for tao-ds-oss-develop-50-2njfx-xnrzg-2x92t; tao-ds-oss-develop-50-2njfx-xnrzg-2x92t was marked offline: Connection was broken

Expected: the build_docker_image stage builds and pushes the image, and the pipeline finishes green.
Observed: the agent pod tao-ds-oss-develop-50-2njfx-xnrzg-2x92t was marked offline mid-stage (Connection was broken), the durable-task sh step could not even create its control directory (FilePath.mkdirs → ClosedChannelException), and the build aborted. The retry wrapper around the stage could not recover because the agent itself was gone, not just the command.

This failure is not attributable to PR #31. That PR changes exactly two files — a one-line change in nvidia_tao_ds/data_analytics/scripts/kpi_analyze.py and a new 116-line regression test tests/test_kpi_analyze_regression.py. Both are exercised by the test stage that passed cleanly, and neither participates in Docker image construction. The disconnect occurred roughly 20 minutes later, in a stage that only pulls and flattens the prebuilt base image nvcr.io/nvstaging/tao/data_services_base_image.

Proposed Design

There is no source change in this repository that can fix this failure, which is why no fix PR accompanies this issue. The relevant control points all live outside the repo:

• The build_docker_image / docker_daemon_setup stage definitions, the retry wrapper, and the "pre-flattening" logic are defined in the internal Jenkins pipeline (CI_SERVER); there is no Jenkinsfile or equivalent under version control here (.github/workflows/ contains only GitHub-side workflows: blossom-ci.yml, static-tests.yml, dco.yml, pr-title.yml, secret-scan.yml, cherry-pick.yml, pr-reminder.yml).
• release/docker/Dockerfile.release only declares FROM nvcr.io/nvstaging/tao/data_services_base_image@sha256:${X86_DIGEST} AS base-amd64; the flatten step that crashed is driven by the Jenkins pipeline, not by this Dockerfile.

Recommended follow-up for the CI/infra owners:

1. Inspect the agent pod's termination reason. Connection was broken immediately after starting a multi-GB base-image pull/flatten is the classic signature of a Kubernetes eviction — OOM-kill or ephemeral-storage/disk-pressure eviction on the node. Check the pod's lastState.terminated.reason and node events for Evicted / OOMKilled around 2026-08-19T23:34–23:35Z.
2. Raise the agent pod's memory and ephemeral-storage requests/limits for the docker container used by build_docker_image, sized for flattening the data_services_base_image layer set.
3. Make the stage resilient to agent loss. The existing retry only re-runs the step on the same agent; once the agent is offline the retry is useless. Consider wrapping the stage so a lost agent reschedules onto a fresh pod, or add an explicit node-level retry.
4. If the disconnect turns out to be network-only, review the Jenkins websocket/remoting idle-timeout and keepalive settings between controller and agents.

Re-running /build on PR #31 is the immediate unblock — an agent disconnect of this kind is not deterministic and typically passes on a fresh pod.

Alternatives Considered

• Modify release/docker/Dockerfile.release to reduce image size / layer count. Rejected: the crash happened while flattening the base image, before this Dockerfile's own build steps, so changes here would not address the cause; it would be a speculative edit to unrelated release tooling.
• Quarantine or retry a test. Not applicable — zero tests failed (448 passed, 23 skipped).
• Pin/adjust dependencies. Rejected: the ERROR: pip's dependency resolver ... lines in the log are pre-existing, non-fatal advisory warnings from the base image (pip still reported Successfully installed), and they appear long before — and independently of — the agent disconnect. Similarly Failed to import SAM3. Error: No module named 'sam3' is a benign optional-import notice emitted during a test that passed.

Category

infra_flake

Reproduction Steps

1. Trigger Blossom-CI on PR [TAO-2319][Bugfix] Add class_name column to kpi_calc.csv output #31 (head 45efa90) via a /build comment — run 32313929250 is the resulting workflow_dispatch build.
2. Observe the mirrored Jenkins log in the "Upload log" job: the test and SonarQube stages pass, then the build_docker_image stage aborts with AgentOfflineException for agent tao-ds-oss-develop-50-2njfx-xnrzg-2x92t. Not deterministically reproducible — it depends on agent pod health.

Environment

• Repo: NVIDIA-TAO/tao-data-services
• Python: 3.12 (build container; setup.py classifiers list 3.8)
• Package version: nvidia-tao-core==7.0.0 installed in the CI image
• Failing run: 32313929250 (https://github.com/NVIDIA-TAO/tao-data-services/actions/runs/32313929250) on b9790671aedba93277621feee08cef8bd93396de (dispatch commit on main); code under test: PR [TAO-2319][Bugfix] Add class_name column to kpi_calc.csv output #31 @ 45efa90188907481e3fb120022181fc42b0d6e85
• Detected via: upload_log_scan
• Jenkins agent: tao-ds-oss-develop-50-2njfx-xnrzg-2x92t

Debug Output

[Pipeline] { (build_docker_image)
[Pipeline] retry
[Pipeline] {
[Pipeline] echo
Building Docker image: ****
[Pipeline] withDockerRegistry
Executing sh script inside container docker of pod tao-ds-oss-develop-50-2njfx-xnrzg-2x92t
Executing command: "docker" "login" "-u" "$oauthtoken" "-p" ******** "https://nvcr.io"
Login Succeeded
[Pipeline] {
[Pipeline] echo
Pre-flattening base image **** -> tao-flat-base-amd64:local (stage base-amd64)
[Pipeline] sh
[Pipeline] }
[Pipeline] // withDockerRegistry
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
ERROR: Execution failed
java.nio.channels.ClosedChannelException
	at jenkins.agents.WebSocketAgents$Session.closed(WebSocketAgents.java:160)
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to tao-ds-oss-develop-50-2njfx-xnrzg-2x92t
		at hudson.remoting.Channel.call(Channel.java:1111)
		at hudson.FilePath.act(FilePath.java:1228)
		at hudson.FilePath.mkdirs(FilePath.java:1408)
		at org.jenkinsci.plugins.durabletask.FileMonitoringTask$FileMonitoringController.setupControlDir(FileMonitoringTask.java:313)
		at org.jenkinsci.plugins.durabletask.BourneShellScript$ShellController.<init>(BourneShellScript.java:291)

org.jenkinsci.plugins.workflow.support.steps.AgentOfflineException: Unable to create live FilePath for tao-ds-oss-develop-50-2njfx-xnrzg-2x92t; tao-ds-oss-develop-50-2njfx-xnrzg-2x92t was marked offline: Connection was broken

Logs

# Earliest symptom of the disconnect — surfaced mid-SonarQube, ~90s before the fatal abort:
23:34:41.959 INFO  Load/download plugins
Cannot contact tao-ds-oss-develop-50-2njfx-xnrzg-2x92t: hudson.remoting.RequestAbortedException: java.nio.channels.ClosedChannelException
23:34:59.956 INFO  Load/download plugins (done) | time=17998ms

# Evidence that everything owned by this repo passed before the abort:
========== 448 passed, 23 skipped, 112 warnings in 954.60s (0:15:54) ===========
**** analysis completed successfully

# Benign, pre-existing base-image advisories (NOT the cause):
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed.
nvidia-eff 0.6.6 requires black==22.3.0, which is not installed.
google-genai 2.12.1 requires httpx<1.0.0,>=0.28.1, but you have httpx 0.27.2 which is incompatible.
Successfully installed Flask-3.1.1 ... nvidia-tao-core-7.0.0 ...
Failed to import SAM3. Error: No module named 'sam3'

Checklist

• I confirmed this failure is reproducible (required) — confirmed present in run 32313929250; note that agent-disconnect flakes are non-deterministic by nature
• I searched existing issues and this is not a duplicate (required)

Suggested Labels

VRDC, nightly-ci, auto-diagnosed, ci-failure, needs-triage

---

Suggested Labels (apply manually after triage)

VRDC, nightly-ci, auto-diagnosed, ci-failure, needs-triage

---

_{🤖 Filed automatically by AIRE from Blossom-CI run 32313929250 (PR #31). Detected via upload_log_scan of the mirrored Jenkins log — the GitHub run conclusion was success.}